DOCUMENT  RESUME 


ED  472  223 


IR  058  523 


TITLE 

INSTITUTION 

ISBN 

PUB  DATE 
NOTE 


AVAILABLE  FROM 


PUB  TYPE 
EDRS  PRICE 
DESCRIPTORS 


IDENTIFIERS 


Measures  for  Electronic  Resources  (E-Metrics).  Complete  Set. 
Association  of  Research  Libraries,  Washington,  DC. 
ISBN-0-918006-94-5 
2002-00-00 

498p.;  Includes  a CD-ROM  (not  available  from  ERIC)  titled  "E- 
Metrics  Instructional  Modules"  containing  the  following  four 
PowerPoint  presentations:  The  ARL  New  Measures  Initiative — 
The  E-Metrics  Project:  The  Importance  of  Network  Measures,  and 
Statistics;  Preparing  YOUR  Library  to  Collect,  Report, 
Analyze,  and  Use  the  Statistics  and  Measures;  Recommended 
Statistics  and  Measures  for  Library  Networked  Services;  and 
The  ARL  E-Metrics  Instructional  Modules. 

Association  of  Research  Libraries,  21  Dupont  Circle,  NW, 

Suite  800,  Washington,  D.C.  20036  ($100  for  all  five  parts; 
$25  for  each  individual  part) . Tel:  202-296-2296;  Fax:  202- 
872-0884;  e-mail:  pubs0arl.org;  Web  site:  http:// 
www . arl . or g/stats/newmeas /erne tries /index . html . 

Collected  Works  - General  (020) 

EDRS  Price  MF02/PC20  Plus  Postage. 

Academic  Libraries;  *Data  Collection;  Evaluation  Criteria; 
Evaluation  Methods;  Higher  Education;  Information  Technology; 
Library  Collections;  Library  Development;  ^Measurement 
Techniques;  *Nonprint  Media;  ^Research  Libraries; 

^Statistical  Data 

^Association  of  Research  Libraries;  ^Electronic  Resources 


ABSTRACT 
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linking  the  use  of  electronic  resources  to  institutional  outcomes.  Part  1 of 
this  publication  contains  the  background  of  the  project  and  the  report  from 
Phase  One,  which  documents  an  analysis  of  current  practice  among  ARL  member 
libraries  regarding  the  collection  of  information  on  electronic  resources. 
Part  2 contains  the  report  from  Phase  Two,  which  documents  the  process  by 
which  a set  of  measures  was  field-tested  by  project  participants,  the  project 
investigators1  and  participants1  work  with  vendor  statistics,  and  the 
resulting  recommendations  from  the  project  investigators  about  which 
statistics  and  measures  should  be  collected.  Part  2 also  includes  a 
compilation  of  results  from  the  E-Metrics  vendor  statistics  field  test.  Part 
3 provides  an  instructional  module  for  institutions  to  use  to  train  their 
staff  to  collect  the  statistics  and  measures  recommended  by  the 
investigators,  and  Part  4 is  the  data  collection  manual  for  the  recommended 
statistics  and  measures.  Part  5 includes  two  papers  regarding  the  linkage  of 
measures  to  institutional  outcomes,  one  by  the  project  investigators  and 
another,  a commissioned  paper  on  the  analysis  of  accreditation  standards  of 
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In  January  1999,  members  of  the  Association  of  Research  Libraries  Statistics  and 
Measurement  Committee,  the  Leadership  and  Management  Committee,  and  other  interested 
member  leaders,  gathered  in  a retreat  setting  to  discuss  what  ARL  could  do  to  assist  members  in 
developing  new  measures  that  better  describe  research  libraries  and  their  services.  Those 
attending  the  retreat  addressed  a set  of  questions  regarding  the  data  needed  to  describe  research 
libraries  in  today’s  environment,  the  need  for  new  measures,  and  the  means  by  which  useful  data 
and  measurement  tools  could  be  developed.  The  result  of  this  meeting  was  the  establishment  of 
ARL’s  New  Measures  Initiative. 

One  of  the  high  priority  needs  expressed  at  the  retreat  was  for  measures  in  the  area  of 
electronic  resources.  Many  libraries  were  expending  an  increasing  proportion  of  their  materials 
budgets  for  electronic  information  and  were  looking  for  consistent  and  reliable  tools  by  which 
they  might  measure  their  investment  in  those  resources  and  for  methodologies  that  could  help 
determine  what  difference  those  investments  made  to  their  user  community.  A planning  session 
was  held  in  February  2000  to  determine  how  to  structure  a project  that  would  address  ARL 
member  interests  surrounding  usage  measures  for  electronic  information  resources.  A number  of 
members  agreed  to  self-fund  a project  and  Sherrie  Schmidt,  Dean  of  University  Libraries, 

Arizona  State  University,  and  Rush  Miller,  University  Librarian  and  Director,  University  of 
Pittsburgh  Libraries,  served  as  the  project’s  co-chairs.  In  response  to  a study  proposal  by  Charles 
R.  McClure,  ARL  subsequently  contracted  with  Florida  State  University’s  Information  Use 
Management  and  Policy  Institute  to  conduct  the  project. 

The  E-Metrics  study  was  designed  as  an  18-month  project  in  three  phases:  an  inventory 
of  what  libraries  were  already  doing  about  data  collection  for  electronic  resources  and  an 
identification  of  any  libraries  that  could  provide  best  practice;  identifying  and  testing  data 
elements  that  could  be  collected  and  used  as  measures  for  electronic  resources  for  both  trends  and 
benchmarking;  and  linking  the  use  of  electronic  resources  to  institutional  outcomes.  The  scope  of 
the  E- Metrics  project  significantly  expanded  from  the  original  prospectus  with  the  (1)  amount  of 
work  and  the  complications  that  developed  as  a result  of  scheduling  meetings  and  data  collection 
activities  with  vendors,  (2)  coordinating  statistics  development  with  other  organizations  (ICOLC, 
NISO,  PALS,  NCLIS,  etc.),  and  (3)  the  level  of  effort  required  by  everyone  involved  with  the 
project  to  complete  the  field  testing.  At  the  conclusion  of  the  field  testing,  the  investigators 
provided  a set  of  recommendations  to  ARL  regarding  the  collection  of  data  for  electronic 
resources. 

The  formal,  funded  E-Metrics  project  has  been  completed.  The  Phase  I report  on  the 
identification  of  current  activities  being  undertaken  in  ARL  libraries  to  support  data  collection  for 
electronic  resources  was  issued  in  November  2000.  Phase  II  identified  a set  of  statistics  and 
measures  that  can  be  used  to  describe  electronic  resources  in  ARL  libraries  and  was  issued  in 
October  2001 . Phase  III  provided  a document  describing  a project  to  link  electronic  measures  to 
institutional  goals  and  objectives  and  was  finalized  at  the  end  of  2001.  Project  documents  can  be 
found  at  the  project’s  Web  site  <http://www.arl.org/stats/newmeas/emetrics/index.html>  and  are 
included  in  this  printed  publication. 


0 


7 


Part  1 of  this  publication  contains  background  of  the  project  and  the  report  from  Phase  I, 
which  documents  an  analysis  of  current  practice  among  ARL  member  libraries  regarding  the 
collection  of  information  on  electronic  resources.  Part  2 contains  the  report  from  Phase  II,  which 
documents  the  process  by  which  a set  of  measures  was  field-tested  by  project  participants,  the 
project  investigators’  and  participants’  work  with  vendor  statistics,  and  the  resulting 
recommendations  from  the  project  investigators  about  which  statistics  and  measures  should  be 
collected.  In  the  report  the  investigators  provide  a data  collection  manual  that  includes  specific 
definitions  on  the  statistics  and  guidance  on  how  to  structure  some  internal  operations  in  order  to 
collect  the  recommended  statistics.  Part  2 also  includes  a compilation  of  the  results  from  the  E- 
Metrics  vendor  statistics  field  test.  Part  3 provides  an  instructional  module  for  institutions  to  use 
to  train  their  staff  to  collect  the  statistics  and  measures  recommended  by  the  investigators  and 
Part  4 is  the  data  collection  manual  for  the  recommended  statistics  and  measures.  Part  5 includes 
two  papers  regarding  the  linkage  of  measures  to  institutional  outcomes,  one  by  the  project 
investigators  and  another  a commissioned  paper  on  the  analysis  of  accreditation  standards  of 
higher  education  commissions. 

The  complete  set  of  documents  represents  the  investigators'  final  reports  to  the 
Association  of  Research  Libraries  and  were  accepted  as  completion  of  the  contract.  The  project 
participants  reviewed  the  reports  from  the  investigators  in  October  2001  and  have  recommended 
to  the  ARL  Statistics  and  Measurement  Committee  that  further  work  in  testing  the  suggested 
measures  be  conducted.  ARL  is  continuing  to  review  the  other  recommendations  in  the  reports. 
Activities  being  undertaken  to  continue  the  work  of  the  project  include: 

• Continued  testing  of  the  proposed  measures  by  project  participants  and  other  interested  ARL 
members  to  determine  the  level  of  effort  needed  to  collect  the  data  and  to  determine  if  there  is 
enough  consistency  of  results  to  be  able  to  extend  the  collection  effort  to  the  whole  ARL 
community. 

• Continued  work  with  the  project’s  Task  Force  on  Statistics  from  Vendor-Based  Database 
Products  to  identify  a small  core  set  of  data  that  vendors  can  provide  to  libraries  with 
consistency. 

• Collaboration  when  possible  with  other  national  and  international  efforts  in  the  area  of 
electronic  resources  and  vendor/publisher  statistics,  including  the  option  of  supporting  a 
multi-agency  organization  tasked  to  develop  an  achievable  and  widely  supported  common 
code  of  practice  for  vendor-based  online  usage  statistics. 

• Publicizing  the  work  of  the  project  within  the  ARL  and  academic  library  community. 

Julia  Blixrud 
June  2002 
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E-Metrics:  Measures  for  Electronic  Resources 


by 

Rush  Miller 

University  Librarian  and  Director 
University  of  Pittsburgh 

and 

Sherrie  Schmidt 

Dean  of  University  Libraries 
Arizona  State  University 

Keynote  delivered  at  the  4th  Northumbria  International  Conference  on  Performance 
Measurement  in  Libraries  and  Information  Centers 

Abstract:  A major  problem  facing  research  libraries  today  is  the  lack  of  data  about  electronic  resources 
and  services.  Problems  and  challenges  in  collecting  and  analyzing  such  data  are  many  and  obvious,  including:  there 
is  a lack  of  clear  and  consistent  definition  of  data  elements;  vendors  do  not  “count”  things  in  the  same  manner  as  one 
another;  membership  in  a consortium  can  skew  the  statistics  of  the  individual  libraries  in  that  consortium;  libraries 
structure  themselves  differently  in  regard  to  electronic  resources,  making  data  gathering  difficult;  libraries  do  not 
control  access  to  and  use  of  important  data  about  vendor-supplied  resources;  and  the  nature  of  electronic  resources  is 
changing  rapidly  and,  therefore,  data  elements  are  shifting.  The  E-Metrics  project,  one  of  the  ARL  New  Measures 
Initiatives,  is  an  effort  to  explore  the  feasibility  of  defining  and  collecting  data  on  the  use  and  value  of  electronic 
resources.  ARL  has  experience  in  tracking  expenditures  on  electronic  resources  through  the  ARL  Supplementary 
Statistics,  but  there  is  a widely  held  recognition  that  more  work  needs  to  be  done  in  this  area.  A group  of  24  ARL 
libraries  funded  and  are  participating  in  the  ARL  E-Metrics  Project  from  May  2000  to  December  2001.  The  project 
is  under  contract  with  Florida  State  University’s  Information  Use  Management  and  Policy  Institute  and  is  directed 
by  Wonsik  “Jeff’  Shim,  Charles  R.  McClure,  and  John  Carlo  Bertot  under  the  leadership  of  project  co-chairs  Sherrie 
Schmidt  (Dean  of  University  Libraries,  Arizona  State  University  Library)  and  Rush  Miller  (University  Librarian  and 
Director,  University  of  Pittsburgh).  This  paper  details  the  rationale  and  context  for  this  project;  it  describes  the 
issues  identified,  the  lessons  learned,  and  the  possibilities  and  challenges  that  this  set  of  issues  brings  to  the  research 
library  community. 
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The  research  library  today  can  be  described  as  a “hybrid”  library:  a library  in  transition  from  a focus  on 
print-based  collections  and  services  to  an  emphasis  on  electronic,  or  digital,  information  resources  and  services.  The 
quickening  pace  of  change  in  this  field  is  evident  in  the  supplemental  statistics  data  gathered  by  the  Association  of 
Research  Libraries  (ARL,  200 1 ). 1 The  percentage  of  acquisitions  dollars  that  ARL  member  libraries  devote  to 
electronic  resources  has  risen  from  3.6%  in  1992-93  to  12.9%  in  1999-2000.  Nine  libraries  spent  more  than  20%  of 
their  materials  budget  on  electronic  or  digital  materials  and  five  libraries  spent  more  than  $2  million  on  such 
resources  in  1999-2000,  with  University  of  Pittsburgh  being  at  the  top  of  the  list  spending  $2, 163,220. 2 One 
hundred  and  five  ARL  libraries  reported  spending  a total  of  almost  $100  million  on  electronic  resources  out  of  their 
materials  expenditures  budget.  The  cost  of  mounting  digital  information  resources  is  far  higher  when  infrastructure 
and  personnel  costs  are  factored  into  the  picture.  Clearly,  the  total  expenditures  related  to  electronic  resources  and 
services  within  ARL  libraries  would  be  measured  in  the  hundreds  of  millions  of  dollars  if  it  could  be  counted 
accurately  and  consistently. 

That,  of  course,  is  the  problem.  While  libraries,  particularly  ARL  libraries,  have  60  years  of  consistently 
defined  and  collected  statistics  related  to  budgets,  collections,  services,  and  personnel,3  no  such  data  is  available  for 
the  electronic  resources  that  are  becoming  ever  more  important.  Problems  and  challenges  in  collecting  and  analyzing 
such  data  are  many  and  obvious,  including:  there  is  a lack  of  clear  and  consistent  definition  of  data  elements; 
vendors  do  not  “count”  things  in  the  same  manner  as  one  another;  membership  in  a consortium  can  skew  the 
statistics  of  the  individual  libraries  in  that  consortium;  libraries  structure  themselves  differently  in  regard  to 
electronic  resources,  making  data  gathering  difficult;  libraries  do  not  control  access  to  and  use  of  important  data 
about  vendor-supplied  resources;  and  the  nature  of  electronic  resources  is  changing  rapidly  and,  therefore,  data 
elements  are  shifting.  Even  as  libraries  are  increasing  their  investment  in  electronic  resources  and  the  opportunities 
for  information  management  are  growing  dramatically  with  the  advent  of  the  World  Wide  Web  as  a delivery 
vehicle,  we  know  much  less  about  this  aspect  of  our  collections  and  services  than  the  traditional  ones. 

Questions  related  to  the  measurement  of  digital  resources  and  services  must  be  answered  if  libraries  are  to 
be  accountable  to  their  constituents  and  funders  alike.  Questions  such  as,  “Who  uses  these  resources?”  or  “Are  these 
huge  outlays  of  funds  justified  in  terms  of  use,  or  value  derived  from  use?”  or  “What  difference  do  all  of  these 
resources  make  to  students  and  faculty  in  universities?”  must  be  answered  if  university  administrators,  trustees, 
students,  and  faculty  are  expected  to  support  ever-increasing  levels  of  funding  for  the  acquisition  and  development 
of  these  resources  and  services.  Just  as  important  is  the  need  for  reliable  measures  in  order  to  make  sound  decisions 
about  the  acquisition  or  de-acquisition  of  electronic  resources,  selection  of  what  to  digitize,  and  development  of 
criteria  and  benchmarks  that  can  be  communicated  to  stakeholders. 

ARL  has  been  concerned  with  performance  measurement  issues  since  the  1990s.4  The  ARL  Statistics  and 
Measurement  Committee  and  the  ARL  Leadership  and  Management  Committee  launched  the  New  Measures 

1 Association  of  Research  Libraries,  ARL  Supplementary  Statistics  1999-2000  (Washington,  DC:  Association  of  Research  Libraries,  2001) 

Also  available  at:  <http://www.arl.Org/stats/arlstat/#sup> 

2 Ibid.,  24. 

3 Martha  Kyrillidou,  “Research  Library  Trends:  ARL  Statistics”  in  Journal  of  Academic  Librarianship  (November  2000):  427+ 

, “To  Describe  and  Measure  the  Performance  of  North  American  Research  Libraries”  IFLA  Journal  21  (4)  (2001):  257-263; 

and  Association  of  Research  Libraries,  ARL  Statistics  (annual). 

4 Martha  Kyrillidou  and  Julia  C.  Blixrud,  “Measuring  the  Changing  Library  Environment”  Developing  Indicators  for  Academic  Library 
Performance:  Ratios  from  the  ARL  Statistics  1996-97  and  1997-98.  (Washington,  DC:  Association  of  Research  Libraries,  2001):  1-13. 


2 


11 


Initiative  in  January  1999,  following  a retreat  held  in  Tucson.  The  New  Measures  Initiative  arises  from  two 
challenges  facing  research  libraries:  first,  the  need  to  demonstrate  the  impact  research  libraries  have  on  areas  of 
interest  to  their  host  institutions;  and  second,  the  need  to  respond  to  pressure  to  maximize  resources  through  cost 
containment  and  reallocation,  which  in  turn  requires  the  identification  of  “best  practices”.5  Coming  out  of  the 
Tucson  retreat,  several  representatives  wrote  white  papers  in  areas  of  acknowledged  interest.6  Those  attending  the 
retreat  addressed  a set  of  questions  regarding  the  data  needed  to  describe  research  libraries  in  today’s  environment, 
the  need  for  new  measures,  and  the  means  by  which  useful  data  and  measurement  tools  could  be  developed.  The 
retreat  participants  recognized  that  “any  new  measures  must  (a)  be  consistent  with  organizational  missions,  goals, 
and  objective;  (b)  be  integrated  with  an  institution’s  program  review;  (c)  balance  customer,  stakeholder,  and 
employee  interests  and  needs;  (d)  establish  accountability;  and  (e)  include  the  collection  and  use  of  reliable  and  valid 
data.”  7 

During  1999,  the  library  leaders  engaged  in  this  set  of  activities  decided  that  it  was  not  enough  to  simply 
frame  the  issues — research  libraries  needed  to  move  into  testing  new  methods  and  experimenting  with  specific 
projects.  With  limited  resources  and  many  ideas  to  test  and  implement,  a variety  of  projects  have  emerged  as 
outlined  in  the  annual  ARL  Activities  Report .8  There  are  five  major  projects  that  are  being  pursued  within  the 
Association  under  the  aegis  of  New  Measures.  These  are:  [1]  an  investigation  into  higher  education  outcomes 
assessment,  with  an  examination  of  both  learning  outcomes  9 and  research  outcomes;  [2]  measurement  of  library 
service  quality;10  [3]  cost  studies;  [4]  interlibrary  loan  and  document  delivery  investigation;  and  [5]  an  examination 
of  measures  for  networked  statistics  and  electronic  resources.* 11 

The  examination  of  measures  for  networked  statistics  and  electronic  resources  has  evolved  into  the  ARL  E- 
Metrics  Project.  The  E-Metrics  Project  began  in  February  2000  at  a retreat  in  Scottsdale,  Arizona,  attended  by 
representatives  from  36  ARL  libraries.  This  retreat  focused  on  the  challenges  involved  in  measuring  the 
commitment  to  and  impact  of  electronic  resources  and  services  in  ARL  libraries.  Due  to  his  extensive  funded 
research  in  this  area,12  ARL  employed  a consultant  for  the  meeting — Dr.  Charles  McClure,  Francis  Eppes  Professor 


5 See:  <hnp:// www.arl.org/stats/newmeas/newmeas.html> 

6 Shirley  Baker,  “Ease  and  Breadth  of  Access.”  Available  at:  <http://www.arl  org/stats/program/Access.pdP>:  Brinley  Franklin  and  Danuta 
Nitecki,  “User  Satisfaction  White  Paper.”  Available  at.  <http://www.arl.org/libqual/  geninfo/usersatisfaction.pdP>;  Carolynne  Presser,  “Library 
Impact  on  Research:  A Preliminary  Sketch.”  Available  at:  <http://www.arl  .org/stats/  pro  gram/pre  sser,pdP>:  Cliff  Haka,  Joan  Giesecke,  and 
Jennifer  Gargill,  “The  Measurement  of  Facilities  Utilization  in  ARL  Libraries.”  Available  at:  <http://www.arl.org/stats/program/Facilities.pdP>: 
Paul  Kobulnicky  and  Carla  Stoffle,  “Market  Penetration  in  Research  Libraries.”  Available  at:  <http://www.arl  org / stats/program/M ark et.pdP>; 
Kathryn  J.Deiss,  “Organizational  Capacity  White  Paper.”  Available  at:  ^ttp^/ww'w.arl.org/stats/program/  capacitv.pdP>. 

7 Julia  Blixrud,  "The  Association  of  Research  Libraries  Statistics  and  Measurement  Program:  From  Descriptive  Data  to  Performance  Measures," 
67th  1FLA  Council  and  General  Conference,  August  16-25,  2001.  Available  at:  <http://www.ifla.org/IV/ifla67/papers/034-135e.pdP>. 

8 Association  of  Research  Libraries,  ARL  Activities  Report  2001  (Washington,  DC:  Association  of  Research  Libraries,  2001):  also  see  ARL 
Activities  Report  1999  and  ARL  Activities  Report  2000. 

9 Ken  Smith,  New  Roles  and  Responsibilities  for  the  University  Library:  Advancing  Student  Learning  through  Outcomes  Assessment. 
(Washington,  DC:  Association  of  Research  Libraries,  2000). 

10  Fred  Heath,  Colleen  Cook,  Martha  Kyrillidou,  and  Duane  Webster,  “The  Forging  of  Consensus:  A Methodological  Approach  to  Service 
Quality  Assessment  in  Research  Libraries — The  LibQUAL+™  Experience”  and  Colleen  Cook  and  Bruce  Thompson,  “Scaling  for  the 
LibQUAL+  Instrument.  A Comparison  of  Desired,  Perceived,  and  Minimum  Expectations  Responses  vs.  Perceived  Only,”  forthcoming  in  the  4th 
Northumbria  Proceedings.  For  more  information  and  an  extensive  bibliography  on  LibQUAL+,  see  <http://www.libqual.org/>. 

11  ARL  E-Metrics  Project  homepage:  < http://www.arl.org/stats/newmeas/emetrics/index.html> 

12  Charles  R.  McClure,  Economic  Benefits  and  Impact  from  Public  Libraries  in  the  State  of  Florida.  (Tallahassee,  FL:  Information  Use 
Management  and  Policy  Institute,  2000).  Available  at:  <http://dlis.dos.state.fl.us/bld/Finalreport/>;  John  Carlo  Bertot,  Charles  R.  McClure,  and 
Joe  Ryan,  Statistics  and  Performance  Measures  for  Public  Library  Networked  Services  (Chicago:  ALA,  2000);  John  Carlo  Bertot  and  Charles  R. 
McClure,  Public  Libraries  and  the  Internet  2000.  (Washington,  DC:  NCLIS,  September  2000).  Available  at: 

<http://www'.nclis.gov/statsurv/2000plo.pdP>;  John  Carlo  Bertot  and  Charles  R.  McClure,  Developing  National  Data  Collection  Models  for 


and  Director  of  the  Information  Management  Use  and  Policy  Institute  at  the  School  of  Information  Studies  at  Florida 
State  University.  Rush  Miller,  Hillman  University  Librarian  at  the  University  of  Pittsburgh,  and  Sherrie  Schmidt, 
Dean  of  Libraries  at  Arizona  State  University,  agreed  to  serve  as  project  co-chairs.  Martha  Kyrillidou,  Senior 
Program  Officer  for  Statistics  and  Measurement,  staffs  the  project  for  ARL.  Susan  Jurrow  served  as  facilitator  for 
the  retreat. 


The  Scottsdale  retreat  was  essential  for  defining  the  scope  of  a project  to  be  undertaken,  since  the  project 
was  to  be  self-funded  as  well  as  self-managed  by  libraries  willing  to  put  forth  a significant  commitment  of  money 
and  staff  time.  Prior  to  the  meeting,  attendees  were  asked  to  submit  answers  to  questions  about  their  efforts  to 
measure  the  impact  of  electronic  services  and  resources  and  their  decision-making  process  related  to  these  materials. 
Also,  some  attendees  provided  examples  of  the  statistics  they  were  collecting;  these  examples  reflected  the  lack  of 
consistency  in  current  practices,  as  well  as  the  lack  of  adequate  data  provided  by  vendors.  After  a full  day  of 
intensive  discussions,  a project  began  to  take  shape.  The  group  identified  four  major  areas  that  should  be  explored  in 
the  project: 

1 . Study  of  users  and  uses. 

2.  Cost  and  benefit  analysis. 

3.  Study  of  staff  impact  and  needs. 

4.  Engagement  with  information  providers  and  their  usage  data  services. 

The  project  co-chairs  worked  with  McClure  and  his  staff  to  develop  a project  prospectus.13  In  the 
meantime,  the  level  of  commitment  in  terms  of  the  number  of  ARL  libraries  electing  to  participate  in  this  project 
doubled  initial  expectations,  for  a total  of  24  libraries  agreeing  to  support  and  participate  in  the  project: 


• University  of  Alberta 

• Arizona  State  University 

• Auburn  University 

• University  of  Chicago 

• University  of  Connecticut 

• Cornell  University 

• University  of  Illinois-Chicago 

• University  of  Manitoba 

• University  of  Mary  land- Co  liege  Park 

• University  of  Massachusetts 

• University  of  Nebraska-Lincoln 

• University  of  Notre  Dame 

• University  of  Pennsylvania 


Pennsylvania  State  University 
University  of  Pittsburgh 
Purdue  University 
University  of  Southern  California 
Texas  A&M  University 

Virginia  Polytechnic  Institute  & State  University 
(Virginia  Tech) 

University  of  Western  Ontario 
University  of  Wisconsin-Madison 
Yale  University 
Library  of  Congress 

The  New  York  Public  Library,  the  Research  Libraries 


The  project  was  formalized  as  the  E-Metrics  Project  and  a formal  contract  was  negotiated  with  the 
Information  Use  Management  and  Policy  Institute  at  Florida  State  University  to  accomplish  the  three  phases  of 
deliverables  outlined  below: 


Public  Library  Network  Statistics  and  Performance  Measures  (funded  by  the  Institute  of  Museum  and  Library  Services,  August  20,  2000). 
Available  at:  < http://slis-rwo.lis  fsu.edu/~-cmcclure/IMLS.html> 

13  Charles  R.  McClure,  "Developing  Statistics  and  Performance  Measures  to  Describe  Electronic  Information  Services  and  Resources  in  ARL 
Libraries."  Available  at:  < http://www.arl.org/stats/newmeas/emetrics/phaserwoappendix.pdf>. 
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A knowledge  inventory  of  ARL  libraries  and  the  organization  of  a 
Working  Group  on  Database  Vendor  Statistics. 

Statistics  and  performance  measures  to  collect  and  analyze  data  collected  within  libraries 
or  provided  by  vendors. 

An  outline  of  a proposal  for  measuring  outcomes  of  electronic  resources,  to  be  funded 
separately. 

The  Phase  One  Report14  was  submitted  to  ARL  on  7 November  2000.  In  this  report,  McClure  and  the 
Institute  staff  report  their  findings  from  their  collection  of  data  related  to  the  current  state-of-the-art  within  ARL 
libraries  in  measuring  electronic  information  resources  and  services.  Their  data  was  gathered  using  survey 
questionnaires  as  well  as  site  visits  to  several  libraries  that  were  considered  advanced  in  this  area  after  an  analysis  of 
the  surveys. 

The  survey  responses  revealed  a wide  range  of  data  collection  and  use  activities  among  the  24  project 
participants.  The  most  consistently  collected  and  used  data  related  to  patron-accessible  resources  and  costs.  Data 
related  to  use  and  users  was  collected  less  often  since  vendors  provide  much  of  the  data  collected  and  it  is  not  kept 
in-house.  Collected  data  was  used  primarily  when  making  acquisitions  decisions.  Not  surprisingly,  the  largest 
impediment  to  survey  respondents  lay  in  the  lack  of  consistent  and  comparable  statistics  from  database  vendors. 


Phase  One: 


Phase  Two: 


Phase  Three: 


Site  visits  were  conducted  at  Virginia  Tech,  the  University  of  Pennsylvania,  Yale  University,  and  the  New 
York  Public  Library.  These  visits  documented  current  practices  and  clarified  survey  responses.  Again  it  was  clear 
that  a lack  of  standardized  reporting  practices  makes  it  difficult  to  collect  and  analyze  data. 


Another  aspect  of  Phase  One  was  the  organization  of  a working  group  to  deal  with  vendor-supplied 
statistics.  This  working  group  met  with  12  major  vendors  for  ARL  libraries  in  order  to  explore  issues  related  to  the 
perceived  lack  of  consistency  in  vendor  statistics  and  to  solicit  vendors’  assistance  in  developing  and  field-testing 
standard  data  elements.  The  vendors  who  accepted  the  invitation  to  participate  in  the  meeting  include: 


Academic  Press/IDEAL 

Bell  & Howell 

EBSCO 

Elsevier/ScienceDirect 

Gale  Group 

JSTOR 

Lexis-Nexis 

netLibrary 

OCLC/FirstSearch 

Ovid 

SilverPlatter 

As  the  project  entered  Phase  Two,  the  focus  shifted  to  the  definition  and  testing  of  data  elements.  Without 
solid  and  comparable  data,  measurement  would  be  less  helpful  and  meaningful  in  the  long  run.  It  was  becoming 
clear  that  the  project  framers  had  underestimated  the  complexity  of  the  issues  and  challenges.  It  also  became  clear 
that  this  project  was  one  of  many  being  undertaken  in  the  United  States  and  in  other  countries  to  accomplish  similar 
if  not  the  same  goals. 


A number  of  projects  designed  to  improve  the  availability  of  consistent  and  comparable  statistical  data 
about  electronic  resources  and  services  have  been  undertaken  over  the  past  several  years.  All  of  these  projects  are 


14  Wonsik  “Jeff”  Shim,  Charles  R.  McClure,  and  John  Carlo  Bertot,  ARL  E-Metrics  Project:  Developing  Statistics  and  Performance  Measures  to 
Describe  Electronic  Information  Services  and  Resources  in  ARL  Libraries:  Phase  One  Report.  (Tallahassee,  FL:  Information  Use  Management 
and  Policy  Institute,  2000).  Available  at:  <http://www.arl.org/stats/newmeas/emetrics/phaseone.pdf> 


related,  in  one  way  or  another,  to  the  E-Metrics  Project.  However,  none  of  them  duplicated  the  ARL  effort  in  terms 
of  goals  and  timeframes.  The  project  co-chairs  undertook  close  communication  links  and  collaboration  with  these 
projects.  These  projects  are: 

• European  Commission  EQUINOX  Project15 

• Publishing  and  Library  Solutions  Committee  (PALS)  Working  Group  on  Online  Vendor  Usage  Statistics 
(UK) 

• International  Coalition  of  Library  Consortia  (ICOLC)  review  of  ICOLC  Guidelines  for  Statistical  Measures 
of  Usage  of  Web- based  Indexed,  Abstracted,  and  Full-Text  Resources 

• National  Commission  on  Libraries  and  Information  Science  (NCLIS)  project  to  standardize  online  database 
usage  statistics  and  reporting  mechanisms  (public  libraries) 

• Institute  of  Museum  and  Library  Services  (IMLS)  project  to  develop  national  network  online  statistics  and 
performance  measures  for  public  libraries 

• Council  on  Library  and  Information  Resources  (CLIR)  report  by  consultant  Judy  Luther  related  to  network 
statistics16 

• NISO  Forum  on  Performance  Measures  and  Statistics17 

During  Phase  Two  of  the  project,  statistical  data  elements  were  discussed  within  the  Vendor  Statistics 
Working  Group  and  with  participants  at  various  meetings  held  at  CNI,  ALA,  and  other  meeting  opportunities.  The 
consultants  worked  with  participants  to  develop  a set  of  measures  to  be  tested  in  the  field.  These  included  statistical 
elements  from  vendors — worked  out  as  a separate  trial  with  12  vendors — and  internal  library  statistics  to  be 
collected  by  library  staff. 


A total  of  18  measures  were  agreed  upon  for  adoption  as  a field  test.  These  elements  were  grouped  into 
categories  and  included: 

1.  Information  Content  This  category  includes  elements  such  as  the  number  of  electronic  full-text 
journals  or  reference  sources  to  which  a library  subscribes.  It  also  includes  virtual  “visits”  to  the 
library’s  electronic  resources  and  the  percentage  of  all  monographs  represented  by  electronic  books, 
among  other  elements. 

2.  Information  Services.  These  elements  measure  usage  of  library  digital  collections  as  well  as  the 
percentage  of  reference  and  other  transactions  that  are  digitally  based. 

3.  Technical  Infrastructure.  Technical  infrastructure  is  measured  in  terms  of  the  cost  of  digital 
collections  along  with  support  costs  and  management  information,  such  as  the  expenditures  for 
electronic  journals  and  books  and  other  components. 


An  effort  to  field-test  vendor  statistics  in  selected  libraries  was  also  underway.  This  effort  was  designed  not 
only  to  collect  and  analyze  data  elements  that  are  agreed  upon  and  consistent  with  the  ICOLC  Guidelines,18  but  to 
gather  information  related  to  the  vendor’s  definition  and  compilation  of  these  data.  Judging  from  the  work  so  far, 
vendors  have  varying  methodologies  and  internal  processes,  which  affect  the  consistency  and  standardization  of  data 


13  EQUINOX  homepage:  <http://equinox.dcu. ie/index.html> 

16  Judy  Luther,  White  Paper  on  Electronic  Journal  Usage  Statistics.  (Washington,  DC:  Council  on  Library  and  Information  Resources,  2000). 

17  National  Information  Standards  Organization  homepage:  <http://www.niso.org/stats-rpthtml> 

18  "Guidelines  for  Statistical  Measures  of  Usage  of  Web-based  Information  Resources,"  (International  Coalition  of  Library  Consortia,  2001). 
Available  at:  < http://www.library.yale.edu/consortia/2001webstats.htm>. 
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provided.  Each  vendor  defines  a search  and  retrieval  set  differently,  which  dramatically  affects  the  statistics 
provided.  It  is  safe  to  say  that,  until  now,  comparing  the  data  from  one  vendor  with  that  of  a second  vendor  was 
unreliable  and  misleading.  One  benefit  of  this  project  will  be  to  assist  vendors  and  libraries  alike  in  standardizing 
data  element  definitions  to  gain  more  consistency  across  the  data. 

Internal  data  elements  were  field-tested  in  13  libraries  (including  the  University  of  Texas,  which  is  not  a 
participant  in  the  project,  but  agreed  to  assist  with  the  field  testing,  as  Sue  Phillips  was  serving  in  a liaison  role 
between  the  ARL  project  and  the  ICOLC  revision  of  the  related  guideline).  Along  with  the  data  itself,  these  libraries 
were  asked  to  track  the  amount  of  effort  expended  in  providing  the  data.  There  was  little  consistency  in  the  number 
of  staff  hours  reported — it  ranged  from  3 to  167  hours.  Much  of  the  variance  can  be  explained  by  the  variability  of 
infrastructure  and  experience  within  ARL  libraries  in  maintaining  data  such  as  these.  Libraries  that  are  already 
engaged  in  collecting  and  analyzing  usage  and  management  data  related  to  electronic  resources  found  it  easier  to 
adapt  to  this  field-test;  those  with  little  history  or  experience  found  it  much  more  difficult  to  comply. 

Libraries  in  the  field  test  were  also  asked  to  analyze  how  useful  they  felt  the  collected  data  would  be  to 
them.  Overall,  libraries  clearly  saw  these  measures  as  good  things  to  have  in  the  absence  of  more  detailed  data. 

The  field-testing  of  these  data  elements  was  critical  to  a better  understanding  of  the  challenges  and  issues 
facing  research  libraries  in  systematizing  e-metrics.  This  kind  of  data  collection  does  not  derive  from  traditional 
library  structures,  such  as  acquisitions,  accounting,  and  cataloging,  or  from  other  information  systems  in  place  in 
libraries.  Few  ARL  libraries  have  a system  in  place  for  managing  electronic  resources,  although  the  number  is 
growing.  Additionally,  many  of  the  definitions  and  procedures  for  collecting  this  data  during  the  field  test  varied 
from  current  practices  within  the  participating  libraries,  although  one  major  outcome  of  the  project  will  be  to 
develop  a more  standardized  mechanism  for  gathering  data.  Defining  changing  concepts  such  as  electronic  books  or 
full-text  retrievals  is  painfully  difficult  and  the  distinctions  among  various  resource  types  can  often  be  arbitrary  and 
fluid.  And,  of  course,  in  ARL  libraries,  electronic  resources  are  often  dispersed  throughout  a large  institution  and  are 
not  centrally  managed,  making  data  difficult  to  collect  centrally. 

The  field  test  allowed  the  project  managers  and  consultants  to  refine  the  data  elements  further.  The  Phase 
Two  report  proposes  a refined  set  of  measures  for  implementation  on  an  ongoing  basis.19  These  elements  include 
measures  of  the  nature  and  size  of  the  digital  resources  available  within  an  institution,  the  cost  of  providing  these 
resources  by  category,  and  the  amount  of  activity  documenting  the  use  of  these  resources.  The  report  from  Phase 
Two  is  available  on  the  Web  and  has  been  distributed  to  all  ARL  member  libraries.  It  includes  a procedures  manual 
that  provides  ARL  libraries  with  definitions  and  techniques  for  collecting  standardized  data  related  to  electronic 
resources;  these  definitions  and  techniques  will  guide  ARL  libraries  in  the  implementation  of  ongoing  data 
collection  relating  to  electronic  resources  measures.  It  is  anticipated  that  these  data  elements  will  not  be  static — as 
the  traditional  ones  have  tended  to  be — but  subject  to  continuous  change.  This  is,  after  all,  the  nature  of  the 
networked  environment. 

From  the  outset  of  the  E-Metrics  Project,  libraries  looked  beyond  the  development  of  metrics  to  the 
development  of  outcomes  measures.  Simple  data  is  not  sufficient  to  answer  the  question,  “What  difference  does  this 
tremendous  outlay  of  resources  make  to  the  users  of  libraries?”  Phase  Three  of  the  project  is  envisioned  to  study  and 
recommend  strategies  and  a framework  for  measuring  outcomes,  i.e.,  assessing  the  impact  and  value  of  electronic 


19  Wonsik  “Jeff’  Shim,  Charles  R.  McClure,  Bruce  T.  Fraser,  John  Carlo  Bertot,  Arif  Dagli,  and  Emily  H Leahy,  Measures  and  Statistics  for 
Research  Library  Networked  Services:  Procedures  and  Issues:  ARL  E-Metrics  Phase  //  Report.  (Washington,  DC:  Association  of  Research 
Libraries,  2001).  Available  at:  <httD://www.arl.org/stats/newnieas/emetrics/phaset\voPreface.pdf> 


resources  on  user  behavior  and  effectiveness.  We  all  want  to  know  what  difference  electronic  resources  make,  not  in 
terms  of  inputs,  but  in  terms  of  outputs.  Some  people  are  asking,  ’’Are  we  determined  to  get  it  right  this  time  in 
terms  of  measuring  important  things  rather  than  just  convenient  things?”  The  answer  is  probably  that  we  always 
wanted  to  get  it  right  and  we  always  did  what  we  thought  was  the  right  thing;  yet,  what  is  right  may  differ  from 
context  to  context.  There  is  often  a scientific  positivism  associated  with  statistics  and  measures  that  can  sometimes 
blind  us  to  the  emerging  context  and  uniqueness  of  specific  environments.  Vice  versa,  one  could  argue  that  too 
much  emphasis  on  the  uniqueness  of  a local  context  fosters  an  isolationist  attitude  that  may  not  be  appropriate  for  a 
highly  interconnected  information  environment  with  global  dimensions  that  are  changing,  shifting,  and  affecting  all 
libraries  in  similar  ways. 

The  consultants  working  on  this  project  have  presented  the  results  of  Phase  Two  with  some  analysis  of  the 
strategy  ARL  might  follow  to  achieve  this  higher  level  of  institutional  outcomes  investigation.  However,  outcomes 
assessment  is  viewed  as  being  a separate  project,  for  which  additional  funding  and  time  will  be  required. 

CONCLUSION 

The  ARL  E-Metrics  Project  is  a key  development  in  the  ongoing  effort  to  quantify  and  better  understand 
the  impact  of  emerging  information  technologies  on  library  collections  and  services.  It  has  provided  the  Association 
with  a new  measurement  model — to  which  individual  libraries  have  committed  significant  resources  and  effort 
beyond  the  Association  structure  and  budget — to  further  develop  and  test  in  Phase  Three  of  the  project. 

It  is  difficult  to  overstate  the  hurdles  encountered  in  carrying  out  what  appeared  at  the  outset  to  be  a rather 
simple  idea — collecting  statistics  on  the  effort  ARL  libraries  are  making  to  mount  electronic  resources  and  services. 
The  problems  of  definition,  reliability,  and  consistency  of  data  provided  by  the  vendor  community  alone  are 
daunting.  But  they  are  matched  equally  by  librarians’  lack  of  agreement  on  what  is  important  to  collect,  how  to 
collect  it,  and  how  to  use  what  is  collected.  Most  libraries  lack  experience  with  the  collection  and  analysis  of  data 
related  to  their  investment  in  electronic  resources.  This  is  a new,  emerging,  and  changing  field  and  these  issues  are 
very  complex  and  difficult  to  get  a handle  on. 

However,  in  less  than  the  two  years  to  which  participants  committed  their  funds  and  support,  the  project  is 
producing  a viable  and  implementable  program  of  data  collection  related  to  electronic  networked  resources  in  ARL 
libraries.  This  accomplishment  is  to  the  credit  of  the  directors  and  staff  of  these  24  libraries;  it  is  also  largely  due  to 
the  expertise  and  hard  work  of  the  director  and  staff  of  the  Information  Use  Management  and  Policy  Institute  at 
Florida  State  University. 

In  developing  e-metrics,  libraries  are  only  part  of  a larger  networked  community  concerned  with  similar 
issues.  Some  libraries  are  concerned  with  the  competition  presented  by  Internet  search  engines,  gateways,  and 
portals.  Some  libraries  feel  the  need  to  demonstrate  large  numbers  of  Web  hits  and  other  e-metrics  to  justify  their 
investment  in  electronic  resources.  Yet,  no  matter  how  large  an  electronic  library  is,  it  is  doubtful  that  it  will  ever 
receive  more  Web  hits  than  popular  search  engines,  gateways,  and  portals  such  as  Yahoo  and  Google.  Libraries, 
though,  have  much  more  valuable  resources  to  offer  than  do  any  Internet  search  engine — it  is  our  challenge  to  try  to 
measure  these  contributions. 
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The  Need  for  Action 

The  rapid  changes — and  advances — in  delivering  information  services  and  resources  electronically  pose 
critically  important  challenges  to  the  Association  of  Research  Libraries  (ARL).  These  new  techniques  in  the  delivery 
of  electronic  and  networked  information  have  left  many  ARL  libraries  with  inadequate  data  to  make  a range  of 
decisions  to  provide  cost-effective  and  high  quality  electronic  services  to  their  users.  Basic  questions  such  as  the 
following  have  yet  to  be  adequately  answered: 

What  methods  can  be  developed  to  determine  who  are  the  users,  what  are  the  uses  of  electronic  information 
services  and  resources  delivered  by  ARL  libraries,  and  what  is  the  frequency  of  that  use? 

How  can  ARL  libraries  obtain  timely,  reliable,  comparable,  and  useful  data  from  database  vendors  that 
describe  uses  and  users  of  the  databases?  And,  how  can  ARL  libraries  affect  the  process  that  determines  the 
types  of  data  they  received  from  the  database  vendors? 

To  what  extent  can  electronic  information  services  and  resources  be  linked  to  a range  of  library  and  higher 
education  outcomes? 

As  demonstrated  at  the  ARL  Scottsdale  meeting  on  New  Measures  in  February  2000,  a number  of  ARL  libraries  are 
working  on  these  and  related  questions.  The  methods  being  developed,  however,  to  answer  these  and  related 
questions  are  not  coordinated  and  are  unlikely  to  be  generalizable  across  the  spectrum  of  ARL  libraries. 

The  study  prospectus  described  here  provides  a three-phased  approach  to  answer  these  and  other  questions 
regarding  statistics  and  performance  measures  in  the  delivery  of  electronic  services  and  resources.  Upon  the 
successful  completion  of  the  study  ARL  libraries  will  have: 

A report  that  summarizes  the  current  “state  of  the  art”  of  best  practices  in  ARL  libraries  for  collecting  and 
reporting  statistics  and  performance  measures  related  to  electronic  resources  and  services.  This  report  will 
include  practical  suggestions  and  techniques  that  will  be  of  use  to  ARL  libraries  for  describing  electronic 
services  and  resource  use  and  users. 

A set  of  statistics  and  performance  measures  that  describe  electronic  information  use,  users,  costs,  and 
staffing.  These  statistics  and  measures  can  assist  ARL  administrators  make  better  decisions  regarding 
deployment,  use,  and  purchase  of  electronic  services. 

A short,  concise  manual  that  describes  possible  procedures  for  how  to  collect,  analyze,  and  report  data  to 
produce  such  statistics  and  performance  measures.  The  manual  will  stress  practical  and  usable  techniques 
that  can  be  used  in  ARL  libraries  as  well  as  identify  key  issues  to  consider  when  using  such  statistics. 

A one-day  workshop  for  interested  ARL  staff  offered  by  members  of  the  study  team  that  will  introduce  the 
manual,  describe  how  best  to  use  it,  and  discuss  issues  that  should  be  considered  in  the  use  of  the  manual. 
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• A process  to  collaborate  with  database  vendors  to  develop,  refine,  and  create  statistics  and  performance 
measures  needed  to  describe  database  use,  users,  and  costs.  The  objective  of  this  process  is  to  ensure  that 
database  vendors  understand  the  need  and  importance  of  ARL  libraries  obtaining  reliable  and  comparable 
use  and  user  data  describing  database  use  and  users. 

• Models  that  link  and  describe  relationships  and  possible  impacts  between  electronic  services  and  resources 
and  higher  education  outcomes.  These  models  will  attempt  to  show  where  and  how  electronic  services  and 
resources  contribute  to  accomplishing  selected  higher  education  outcomes. 

These  products  will  be  a first  set  of  tools,  processes,  and  techniques  that  will  assist  ARL  libraries  (1)  better  meet 
the  needs  of  their  users,  and  (2)  make  better  decisions  regarding  the  purchase,  use,  and  deployment  of  electronic 
services  and  resources. 

Leveraging  Existing  Knowledge 

There  is  a limited  number  of  writings  and  still  less  research  available  regarding  the  development  of  statistics 
and  performance  measures  for  the  academic  networked  environment.  This  prospectus  does  not  provide  an  overview 
of  these  writings  - many  of  which  are  listed  on  a bibliography  on  the  New  Measures  Web  page  at 
<http://www.arl.org/stats/newmeas/e-usage.html>.  Generally,  the  literature  identifies  a range  of  issues  and  problems 
regarding  the  development  of  such  statistics.  The  literature  does  not,  however,  adequately  represent  the  practical 
knowledge  and  current  practice  in  this  area — which  was  identified  at  the  February  28-29,  2000,  ARL  conference  in 
Scottsdale. 

There  is  expertise  outside  the  published  literature  that  can  be  leveraged  by  this  project.  The  author  has  been 
working  to  develop  networked  statistics  and  performance  measures  for  public  libraries  under  a project  funded  by  the 
Institute  for  Museum  and  Library  Services  (IMLS)  <http://www.albanv.edu/-imlsstat/>.  Currently,  a number  of 
statistics  and  measures  are  being  field  tested — some  of  which  would  have  clear  applicability  to  academic  libraries. 
Further,  the  project  has  established  important  contacts  and  meetings  with  a number  of  database  vendors  to  discuss 
the  range  and  types  of  data  that  they  might  provide  to  state  and  public  libraries. 

In  addition,  the  author  is  working  with  the  National  Commission  on  Libraries  and  Information  Science  (NCLIS) 
on  the  project,  Testing  National  Public  Library  Electronic  Use  and  Network  Performance  Measures 
<http://www.nclis.gov/libraries/lsp/statist.html>.  Building  on  the  IMLS  study,  additional  work  is  envisioned  to  meet 
and  collaborate  with  database  vendors  to  determine  the  types  of  data  that  can  be  reasonably  obtained  for  libraries. 
Academic  libraries’  direct  and  formal  involvement  to  participate  in  and  extend  these  efforts  to  address  academic 
library  database  statistical  needs  and  issues  could  build  on  existing  work  and  demonstrate  the  interest  and  concern  of 
the  ARL  community. 


Background 

This  study  prospectus  has  been  under  development  since  fall  1999  when  the  author  discussed  the  possibility  of 
such  a study  with  ARL  staff  and  members  of  the  New  Measures  Initiative 

<http://www.arl.org/stats/newmeas/newmeas.html>.  Given  an  initial  level  of  interest  with  leaders  in  the  New 
Measures  Initiative  and  ARL  staff,  a preliminary  project  prospectus  was  presented  for  discussion  at  a meeting  held 
in  conjunction  with  the  December  1999  Coalition  for  Networked  Information  (CNI)  meeting  in  Phoenix. 

Based  on  discussions  and  interest  from  the  meeting  at  CNI,  McClure  re-drafted  the  prospectus  and  it  was 
distributed  to  a number  of  ARL  members  and  posted  at  the  ARL  Web  site.  Leaders  of  the  New  Measures  Initiative 
with  ARL  staff  organized  a meeting  on  February  28-29,  2000,  to  discuss  the  appropriateness  of  working  in  the  area 
of  developing  statistics  and  performance  measures  for  networked  information  services  and  resources.  Seventy 
participants  discussed  aspects  of  the  project  at  the  meeting  and  provided  guidance  and  suggestions  for  a study.  This 
Study  Prospectus  is  based  on  the  discussions  and  outcomes  from  that  meeting. 
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A Changing  Context 

Clearly,  the  context  for  delivery  of  library  services  and  resources  is  changing  to  an  electronic  environment. 
While  there  has  been  considerable  discussion  about  the  need  and  importance  of  developing  statistics  and 
performance  measures  to  describe  networked  and  electronic  services  and  resources,  a unified  research  strategy  is 
needed  to  move  forward  on  this  topic. 


The  Key  Term 

The  working  definition  of  network-based  information  services  is:  Those  electronic  information  resources  and/or 
services  that  users  access  onsite  in  the  library,  from  their  office,  dorm,  or  home,  or  from  regional/statewide 
networks.  Examples  of  electronic  networked  resources  include:  local,  regional,  or  statewide  library  hosted  or 
authored  Web  sites  or  library-licensed  databases  (e.g.,  Infotrac,  SearchBank,  EbscoHost).  Especially  important  are 
statistics  that  describe  the  use  of  unique  and  oftentimes  interactive  remote  scientific  and  technical  databases. 

Examples  of  electronic  networked  services  include:  provision  of  access  to  networked  services  such  as  email, 
listservs,  online  reference/assistance,  and  training  in  the  use  of  these  resources  and  services.  In  addition,  libraries 
increasingly  provide  interactive  services  such  as  requesting  services  via  online  forms  (interlibrary  loans,  etc.).  It  is 
likely  that  the  term  “networked  information  resources  and  services”  will  continue  to  evolve  as  the  network  evolves 
and  as  the  study  progresses. 


Growth  in  Electronic  Services  and  Resources 

The  move  to  a networked  environment  has  significantly  increased  the  range  of  services  and  resources  that  the 
library  provides  its  users.  The  library  has  become  a 24  hour  a day  access  point  to  information  services  where  users 
obtain  services  and  resources  on  their  terms  and  when  they  want  such  services — oftentimes  not  coming  to  the  library 
physically  nor  interacting  directly  with  library  staff.  The  costs  to  provide  these  networked  services  and  resources, 
however,  can  be  significant.  Librarians’  inability  to  develop  reliable  and  accurate  methods  to  describe  these  services 
and  costs  injures  their  ability  to  make  good  resource  allocation  decisions,  meet  user  needs,  and  develop  strategic 
plans  for  the  development  and  operation  of  electronic  services  and  resources. 

On  an  experiential  basis,  most  ARL  librarians  will  describe  the  use  of  their  networked  information  services  with 
terms  such  as  “exponential  growth”  or  “we  can’t  keep  up  with  demand.”  At  the  same  time,  a number  of  ARL 
libraries  have  also  seen  stagnant  or  declining  statistics  of  traditional  indicators  of  library  service  such  as  turnstile 
counts,  in-house  reference  transactions,  circulation,  etc.  While  there  is  a need  to  develop  new  statistics  such  as 
“virtual  visits,”  “full-text  downloads,”  “electronic  reference  transactions  as  a percentage  of  all  transactions,”  etc., 
there  is  little  agreement  on  how  to  compute  such  statistics  and  measures. 

Example  Possible  Statistics 

The  work  done  by  the  author  and  others  in  the  IMLS  project,  individual  ARL  libraries,  as  well  as  work  under 
way  with  the  NCLIS  study  suggest  that  a number  of  statistics  describing  networked  and  electronic  services  and 
resources  may  be  developed  in  the  following  areas: 

• Count  of  electronic  reference  transactions. 

• Virtual  visits  (sessions)  to  the  library’s  Web  site. 

• Counts  of  high-use  and  low-use  Web  pages. 

• Count  of  sessions  on  specific  databases. 

• IP  addresses  for  sessions  on  specific  databases. 

• Time  per  session  on  specific  databases. 

• Tum-aways  per  time  period  per  specific  database. 

• Primary  use  of  selected  electronic  services  and  resources. 

• Hours  of  user  training  on  electronic  services  by  library  staff. 

• Cost  per  session  on  specific  databases. 

• Count  of  full-text  downloads  per  time  period  per  database. 
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• File  size  of  full-text  downloads  per  time  period  per  database. 

• Count  of  on-site  versus  remote  sessions  per  database. 

These  proposed  statistics  are  illustrative  only.  Yet  to  be  accomplished  is  agreeing  on  definitions  and  data  collection 
methods  to  produce  reliable  and  valid  statistics,  determining  the  degree  to  which  such  statistics  can  also  be 
comparable  across  different  libraries,  and  making  linkages  between  such  statistics  and  higher  educational  outcomes. 
In  short,  considerable  work  has  yet  to  be  done. 


Issues  and  Challenges 

Some  of  the  factors  militating  against  the  development  of  networked  statistics  and  performance  measures 
include  the  following: 

• Librarians  do  not  control  access  to  and  use  of  a range  of  data  that  describe  vendor-supplied  information 
services  and  resources.  Some  vendors  are  unwilling  or  unable  to  provide  the  types  of  statistics  and  use  data 
that  librarians  request.  Statistics  and  measures  for  database  use  and  services,  nonetheless,  are  essential. 

• The  rapidly  changing  nature  of  the  networked  environment  also  affects  the  types  of  services  and  resources 
that  can  be  provided  by  libraries.  As  the  networked  services  change,  new  types  of  statistics  and  measures 
may  be  needed. 

• The  level  of  effort  needed  to  collect,  analyze,  and  report  data  to  produce  statistics  and  performance 
measures  for  the  networked  environment  may  be  greater  than  that  needed  to  produce  more  traditional 
statistics. 

• Sometimes  networked  services  costs  and  use  may  be  difficult  to  “unbundle”  if  the  library  obtains  these 
services  through  a consortium.  Costs  can  either  be  hidden  or  be  extremely  difficult  to  allocate  to  individual 
libraries. 

Librarians  may  be  entering  a period  of  time  where  statistics  and  measures  for  networked  services  may  be  useful  for 
two  to  four  years  and  then  will  have  to  be  re-developed  or  discarded.  Such  an  environment  is  quite  different  than  the 
statistics-collecting  environment  in  which  academic  libraries  previously  existed.  Despite  these  concerns  and  factors, 
ARL  libraries  need  to  move  forward  and  learn  how  best  to  produce  and  use  such  statistics  and  measures  in  this  new 
environment. 

Project  Goals  and  Research  Questions 

One  key  goal  of  this  project  is  to  develop,  test,  and  refine  selected  statistics  and  performance  measures  to 
describe  electronic  services  and  resources  in  ARL  libraries.  A second  goal  is  to  engage  in  a collaborative  effort  with 
selected  database  vendors  to  establish  an  ongoing  means  to  produce  selected  descriptive  statistics  on  database  use, 
users,  and  services.  A third  goal  of  the  project  is  to  develop  a proposal  for  external  funding  to  maintain  the 
development  and  refinement  of  networked  statistics  and  performance  measures.  More  specifically,  the  project  has 
the  following  research  questions: 

• What  existing  techniques  and  approaches  are  being  used  by  ARL  libraries  to  produce  statistics  and 
performance  measures  to  describe  networked  information  services  and  resources?  What  can  be  learned 
from  these  techniques  that  could  be  generalized  to  other  libraries? 

• For  what  purposes  and  for  what  audiences  are  networked  statistics  and  measures  needed? 

• Which  types  of  networked  services  and  resources  should  be  described,  how  should  they  be  defined  and 
operationalized,  and  how  should  the  data  be  collected  to  insure  reliable  and  valid  data? 

• What  performance  or  quality  indicators  are  needed  to  describe  the  impact  and  success  of  such  networked 
services? 
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• How  might  such  statistics  and  measures  be  best  reported? 

• What  linkages  can  be  established  between  these  statistics  and  performance  measures  with  selected 
outcomes  from  higher  education? 

This  list  of  research  questions  suggests  a beginning  perspective  to  direct  project  activities.  These  research  questions 
may  be  revised  as  the  project  proceeds. 

Project  Approach  and  Management 

This  prospectus  assumes  that  a group  of  interested  ARL  libraries  will  work  together  as  a consortium  in 
conjunction  with  a study  team  at  the  Information  Management  Use  and  Policy  Institute  at  Florida  State  University 
<http://www.ii.fsu.edu>  and  with  ARL  staff.  The  members  of  this  group  would  each  agree  to  make  a commitment  of 
$10,000  to  fund  the  project.  The  study  would  begin  May  12,  2000  and  be  completed  December  2001  (20  months). 
ARL  libraries  would  make  their  contribution  to  ARL;  the  Information  Management  Use  and  Policy  Institute  would 
then  enter  into  a contract  with  the  Association  of  Research  Libraries  to  complete  the  study  as  outlined  in  this 
prospectus. 

Participants  in  the  study  may  have  varying  levels  of  direct  involvement  in  the  project.  Some  may  decide  to  be 
active  participants  in  the  project  in  which  they  would  provide  detailed  descriptions  of  their  current  activities 
regarding  the  collection  and  use  of  networked  statistics;  participate  in  reviewing  and  commenting  on  project 
documents  and  reports;  serving  as  field  sites  to  test  and  refine  statistics  and  performance  measures;  meet  with  other 
participants  to  review  project  activities;  organize  meetings  and  handle  logistics  related  to  those  meetings;  provide 
direct  feedback  and  work  with  the  study  team  to  complete  project  goals;  and  engage  in  specific  data 
collection/analysis  activities  related  to  project  activities.  Other  study  participants  may  decide  not  to  participate  in 
such  activities  or  only  in  those  especially  appropriate  to  their  institution. 

In  each  participating  library  there  would  be  one  person  who  would  serve  as  a liaison  and  as  a single  point  of 
contact  to  coordinate  project  activities.  Depending  on  the  level  of  involvement  by  the  library,  this  person  should  be 
prepared  to  contribute  up  to  25%  of  his/her  time  to  the  project  with  some  additional  assistance  from  others  in  the 
library  from  time  to  time. 

All  libraries  participating  in  the  project  may  appoint  a representative  to  serve  on  the  project’s  advisory 
committee  (AC).  This  group  will  provide  feedback  and  suggestions  to  the  study  team  as  needed;  they  will  be  kept 
informed  of  project  activities;  and  they  will  meet  from  time  to  time  (electronically  and  in  person)  to  discuss  project 
progress  and  issues.  The  advisory  committee  will  be  co-chaired  by  Rush  Miller  and  Sherrie  Schmidt.  Miller  and 
Schmidt  will  serve  as  the  single  point  of  contact  with  the  study  team. 

Three  types  of  staff  time  will  be  committed  to  this  project.  First,  the  study  team  at  Florida  State  University  will 
commit  time  and  effort  to  the  project  as  described  later  in  this  prospectus.  Second,  staff  from  participating  ARL 
libraries  will  be  contributing  time  and  effort  to  the  project  as  described  earlier.  Third,  ARL  staff  will  commit  time 
and  effort  to  the  study  by  assisting  in  selected  data  collection  activities,  insuring  effective  communications  with 
ARL  libraries,  providing  background  information  and  resources,  and  maintaining  a Web  site  to  describe  and  update 
project  activities. 

Project  Phases  and  Activities 

Based  on  discussions  at  the  ARL  New  Measures  invitational  conference  held  in  Scottsdale,  AZ,  February 
28-29,  2000  the  following  project  phases  and  activities  have  been  developed.  Note  that  two  or  more  activities  within 
a phase  may  occur  simultaneously.  The  project  activities  related  to  developing  statistics  from  database  vendors  are 
an  ongoing  process  throughout  the  entire  study.  Upon  project  funding  detailed  scheduling  and  tasking  will  occur. 

Phase  I:  Knowledge  Inventory  of  ARL  Libraries  and  organizing  an  ARL  Working  Group  on  Database 
Vendor  Statistics  (May  2000-October  2000) 
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The  two  objectives  of  this  phase  are  to  (1)  identify  and  describe  the  current  state  of  the  art  of  statistics  and 
performance  measures  for  networked  services  and  resources  in  ARL  libraries,  and  (2)  organize  an  ARL 
Working  Group  on  Database  Vendor  Statistics  to  begin  discussions  with  database  vendors. 

Activity  1 of  this  phase  will  be  to  conduct  an  inventory  that  will  survey  all  ARL  libraries  as  to  their  current 
practices,  activities,  statistics,  performance  measures,  data  collection  and  reporting  processes.  After  the  survey 
has  been  completed,  the  study  team  will  conduct  site  visits  at  those  libraries  that  appear  to  have  the  most  useful 
information  and  insights  into  study  topics.  The  inventory  will  stress  data  collection,  statistics,  and  performance 
measures  in  the  following  areas: 

• Users  of  networked  information  services  and  resources,  e.g.,  who  are  the  users  of  specific  types  of 
networked  services  and  resources? 

• Uses  of  networked  information  services  and  resources,  e.g.,  what  are  the  applications  and  uses  of  these 
services  and  resources  by  the  users? 

• Staffing  and  training,  e.g.,  how  have  networked  information  services  and  resources  affected  the  staffing  and 
staff  training  in  libraries? 

• Networked  information  services,  e.g.,  electronic  reference  transactions,  electronic  forms  submission,  etc. 

• Cost  analysis  of  networked  information  services  and  resources,  e.g.,  what  are  the  costs  per  transactions  of 
particular  services  such  as  cost  per  full-text  download? 

• Vendor-based  database  information,  e.g.,  what  statistics  are  being  compiled  by  which  libraries  from  which 
vendors  with  what  information  and  how  are  those  data  defined?  NOTE  that  this  area  is  an  ongoing  project 
effort  that  continues  throughout  all  phases  of  the  project. 

Initially  these  topics  will  provide  a first  priority  for  inventorying  the  current  knowledge  of  ARL  libraries  related 
to  networked  services  and  resources. 

Due  to  a research  award  received  by  study  team  member  Jeff  Shim,  Florida  State  University,  he  will  be 
able  to  conduct  site  visits  during  the  summer  of  2000  to  selected  ARL  libraries  at  no  cost  to  the  project. 

Activity  2 initiates  a process  whereby  an  ARL  Working  Group  on  Database  Vendor  Statistics  can  begin 
discussions  with  selected  vendors  regarding  a plan  to  identify,  collect,  and  report  various  database  statistics. 
Such  statistics  include,  but  are  not  limited  to,  uses  of  the  databases,  frequency  of  use,  time  of  use,  log-ins  to 
particular  titles,  IP  addresses  of  log-ins,  tum-aways,  etc. 

Tasking  for  this  phase  includes  the  establishment  of  an  ARL  Working  Group  on  Database  Vendor 
Statistics:  having  the  group  define  key  issues  and  objectives;  coordinate  activities  with  ICOLC  and  other 
appropriate  groups;  propose  possible  guidelines  and  procedures  for  vendor-produced  database  statistics;  begin 
meeting  with  selected  database  vendors;  and  develop  a process  to  clarify  needs  and  expectations  by  both  the 
vendors  and  the  ARL  Working  Group. 

Products  from  Phase  I:  The  product  from  Phase  I will  be  an  interim  report  that: 

• Describes  the  knowledge  and  best  practices  currently  in  use  by  ARL  libraries  regarding  statistics  and 
performance  measures  for  electronic  information  services  and  resources; 

• Describes  the  range  and  types  of  data  being  obtained  by  ARL  libraries  from  the  various  database  vendors; 

• Describes  key  issues  that  will  need  to  be  resolved  in  producing  statistics  and  performance  measures;  and 

• Presents  a status  report  on  the  activities  of  the  ARL  Working  Group  on  Database  Vendor  Statistics,  its 
objectives,  and  its  next  steps  in  Stage  II. 
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In  addition,  members  of  the  study  team  will  provide  an  Executive  Briefing  to  project  participants  in  conjunction 
with  the  fall  2000  ARL  meeting. 

Phase  II:  Development  of  statistics  and  performance  measures  (November  2000-June  2001) 

The  objective  of  this  stage  is  to  develop  tools,  data  collection  processes,  statistics,  and  performance  measures  to 
describe  services  and  resources  in  the  networked  environment. 

Activity  1 will  develop  an  approach  to  take  the  knowledge  learned  from  Phase  I into  a research  methodology  to 
develop,  define,  and  propose  possible  statistics  and  measures.  In  short,  the  activity  will  produce  a detailed 
tasking  and  methodology  by  which  statistics  and  performance  measures  can  be  developed,  tested,  and  refined. 
This  activity  will  also  propose  data  collection  techniques  and  instruments  for  use  by  participating  libraries  to 
produce  statistics  and  measures  in  each  of  the  areas  identified  in  Phase  I.  The  process  will  draw  heavily  upon 
input  and  advice  (in  an  iterative  fashion)  from  participating  libraries. 

Activity  2 will  be  the  field  testing  of  these  proposed  data  collection  techniques,  statistics,  and  measures.  The 
study  team  anticipates  that  four  to  six  participating  libraries  will  serve  as  field  sites  to  test  the  approaches 
developed  in  Phase  I.  A process  will  be  developed  for  each  library  to  test  some  portion  of  the  proposed  statistics 
and  report  on  the  efficacy  of  the  process  as  well  as  the  statistics  and  measures  themselves. 

Activity  3 will  be  the  analysis  of  the  field  test  results  and  the  writing  of  a short  manual  that  describes  the 
process  for  data  collection,  the  statistics,  and  the  performance  measures.  The  study  team  will  determine  what 
appears  to  be  working  well  in  terms  of  process  and  what  needs  additional  work  and  revision.  Depending  on  the 
results  from  the  field  test,  some  additional  work  and  refinement  on  selected  statistics  and  measures  may  be 
needed  prior  to  writing  the  data  collection,  statistics,  and  measurement  manual. 

Activity  4 of  this  phase  is  to  develop  a model  that  integrates  the  statistics  and  measures  into  (1)  higher 
education  educational  outcomes,  research  outcomes,  and  service  outcomes,  and  (2)  library  educational 
outcomes,  research  outcomes,  and  service  outcomes.  In  fact,  this  phase  will  be  under  development  throughout 
the  entire  phase  as  development  of  statistics  and  measures  must  occur  in  the  broader  context  of  their  purposes 
and  their  relationship  with  higher  education  outcomes.  Findings  from  Phase  I will  inform  the  development  of 
such  models. 

Activity  5 of  this  Stage  is  to  continue  activities  of  the  ARL  Working  Group  on  Database  Vendor  Statistics.  This 
effort  is  ongoing  throughout  Phase  II.  During  this  phase  we  would  expect  that  meetings  between  the  ARL 
working  group  and  the  database  vendors  would  be  moving  toward  agreement  on  data  element  definitions  and 
terms,  to  specific  statistics  and  data  that  can  be  collected,  and  methods  for  reporting  these  data  to  libraries.  The 
study  team  would  expect  to  include  these  data  collection  techniques  in  the  field  test  described  in  activity  3 
above. 

Products  from  Phase  II  include; 

• A written  methodology  to  develop  and  field  test  data  collection  techniques,  statistics,  and  performance 
measures; 

• A short  concise  written  manual  that  describes  how  these  statistics  and  measures  can  be  produced,  that  will 
include: 

introduction  to  the  importance  and  need  for  statistics  and  performance  measures  that  describe 
electronic  services  and  resources; 

data  collection  techniques  and  methods  for  each  of  the  statistics  and  performance  measures; 
issues  to  be  considered  in  using  and  interpreting  these  statistics  and  performance  measures; 
recommendations  for  future  work  in  the  development  of  new  or  refinement  of  existing  statistics  and 
performance  measures;  and 

appendices  of  data  collection  instruments  developed  and  tested  during  the  field  tests; 
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• A written  status  report  on  the  activities  and  accomplishments  of  the  ARL  Working  Group  on  Database 
Vendor  Statistics,  and  issues/next  steps  to  be  addressed;  and 

• A one-day  workshop,  held  in  conjunction  with  ALA  or  an  ARL  meeting  to  present  the  manual  and  discuss 
project  findings  during  the  late  spring  of  2001. 

Phase  III:  Institutionalizing  Statistics  and  Performance  Measures  (July  2001-December  2001) 

The  objective  of  this  phase  is  to  develop  mechanisms  and  processes  that  insure  the  ongoing  development  of 
networked  statistics  and  performance  measures.  This  objective  includes  building  and  promoting  infrastructure 
in  ARL  and  ARL  libraries  to  continue  the  development  and  use  of  such  statistics  and  measures. 

Activity  1 of  this  phase  is  to  develop  a research  proposal  to  obtain  external  funding  to  continue  research  and 
field  testing  related  to  networked  information  statistics  and  measures.  The  study  team  will  work  with  ARL  staff 
and  the  advisory  committee  in  the  development  of  this  proposal  and  the  identification  of  appropriate  funding 
bodies  that  may  be  interested  in  supporting  continued  work  in  this  area.  Specific  research  questions  and 
initiatives  in  this  proposal  will  result  from  findings  and  activities  in  Phase  II. 

Activity  2 of  this  phase  is  to  develop  a number  of  training  modules  and  training  support  systems  (both  in  print 
and  electronically)  that  ARL  and  ARL  libraries  can  use  to  assist  staff  understand  the  importance  of  the  new 
statistics  and  measures  developed  as  part  of  this  project,  as  well  as  help  them  on  a very  practical  level  collect, 
analyze,  and  report  quality  data.  The  study  team  anticipates  developing  and  testing  these  modules  but  assumes 
that  ARL  members  or  ARL  staff  would  be  actively  engaged  in  the  instruction. 

Activity  3 of  this  phase  is  to  continue  refining  and  testing  the  model(s)  that  integrate  the  statistics  and  measures 
into  (1)  higher  education  educational  outcomes,  research  outcomes,  and  service  outcomes,  and  (2)  library 
educational  outcomes,  research  outcomes,  and  service  outcomes.  The  study  team  anticipates  developing  a 
process  to  validate  these  models  and  determine  the  potential  usefulness  of  the  model(s)  to  describe  impacts  from 
networked  information  services  and  resources. 

Activity  4 of  this  phase  will  continue  activities  of  the  ARL  Working  Group  on  Database  Vendor  Statistics.  This 
effort  is  ongoing  throughout  Phase  III.  During  this  phase  we  would  expect  that  meetings  between  the  ARL 
Working  Group  and  the  database  vendors  would  have  reached  agreement  on  data  element  definitions  and  terms, 
on  specific  statistics  and  data  that  can  be  collected,  and  methods  for  reporting  these  data  to  libraries.  The  study 
team  would  expect  to  assist  the  ARL  Working  Group  extend  the  number  of  database  vendors  agreeing  to  the 
guidelines  that  would  have  resulted  for  data  collection  and  statistics. 

Products  from  Phase  III  include: 

• A proposal  that  can  be  submitted  to  potential  fundors  to  continue  research  and  development  on  statistics 
and  performance  measures  in  the  networked  environment  and  their  potential  impacts  on  a range  of  higher 
education  outcomes; 

• A revised  and  updated  description  of  database  statistics  and  performance  measures.  Based  on  the  additional 
activities  of  the  working  group  and  discussions  with  database  vendors  we  would  expect  to  be  able  to  build 
upon  the  database  statistics  developed  in  Phase  II  and  expand  and  refine  them; 

• Instructional  modules  that  can  assist  ARL  and  ARL  member  institutions  train  staff  as  to  the  importance  and 
process  for  collecting  and  analyzing  networked  statistics  and  performance  measures;  and 

• A final  report  of  guidelines  and  issues  yet  to  be  resolved  as  agreed  to  by  ARL  and  participating  database 
vendors  on  the  collection  and  reporting  of  selected  statistics  and  data  elements  from  these  databases. 

It  should  be  recognized  that  some  fine-tuning  of  these  phases  and  activities  may  occur  as  the  project  proceeds.  Such 
fine-tuning  will  be  done  with  the  advice  of  the  advisory  committee. 
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Study  Team  Qualifications 

Charles  R.  McClure  will  serve  as  the  Principal  Investigator  for  the  study.  Detail  on  his  background,  experience, 
and  examples  of  recent  projects  he  has  completed  can  be  found  at  <http://slis-two.lis. fsu.edu/~cmcclure>. 
Information  about  the  Information  Use  Management  and  Policy  Institute  for  which  he  serves  as  director  can  be 
obtained  at  <http://www.ii.fsu.edu>.  He  has  a proven  track  record  of  managing  projects  successfully  and  has  worked 
with  public  libraries  in  a range  of  areas  including  financial  assessment,  planning  and  evaluation  of  information 
services,  information  technology  management,  and  resource  sharing.  Currently,  with  John  Carlo  Bertot,  he  is 
completing  a study  funded  by  the  Institute  for  Museum  and  Library  Services  (IMLS)  that  will  produce  national 
statistics  and  performance  measures  to  describe  public  library  services  and  resources  in  the  networked  environment 
<http://www.albanv.edu/~imlsstat/>. 

John  Carlo  Bertot  is  an  Associate  Professor  in  the  School  of  Information  Science  and  Policy  at  SUNY  Albany. 
He  has  worked  with  and  Charles  R.  McClure  successfully  on  a number  of  library  studies,  including  the  Public 
Libraries  and  the  Internet  national  surveys  from  1994  through  1998,  and  Internet  project  impact  studies  in 
Pennsylvania  {Evaluation  of  the  Online  at  PA  Libraries  Project:  Public  Access  to  the  Internet  through  Public 
Libraries)  and  California  {The  Importance  of  California  Public  Libraries  in  Increasing  Public  Access  to  the 
Internet).  Bertot  and  McClure  have  also  collaborated  on  a number  of  technology  planning  and  evaluation  projects. 
Additional  background  information  on  Bertot’ s experience  and  skills  can  be  found  at 
<http://www.albanv.edu/~icbertot/>. 

Jeff  Shim,  Assistant  Professor  of  Information  Studies  at  Florida  State  University  and  a Research  Associate  at 
the  Information  Use  Management  and  Policy  Institute  will  also  serve  on  the  study  team.  Shim  recently  completed 
his  Ph.D.  from  Rutgers  University  where  his  primary  research  centered  on  an  analysis  of  ARL  statistics.  He  teaches 
in  a range  of  areas  related  to  the  management  of  information  technology.  He  is  especially  knowledgeable  about 
academic  library  statistics,  statistical  techniques  for  describing  services  and  resources,  and  understands  information 
technology-based  services  provision. 

Additionally,  we  expect  that  a number  of  Ph.D.  students  and  other  graduate  students  from  the  School  of 
Information  Studies  at  Florida  State  University  will  be  working  on  the  project  as  research  assistants.  These 
additional  staff  will  be  involved  on  the  project,  oftentimes  with  no  additional  cost  to  the  project. 


Budget  and  Financial  Arrangements 

The  budget  for  this  project  is  $199,990  and  is  summarized  in  Figure  1 . Every  effort  has  been  made  to  keep 
overall  costs  to  a minimum  and  use  budgeted  monies  as  effectively  as  possible. 


Figure  1.  Budget  May  2000-December  2001 
Personnel*  $143,844 

1 . Charles  R.  McClure,  Principal  Investigator 

2.  Jeff  Shim,  Associate  Director 

3.  John  Bertot,  Associate  Director 

4.  Bruce  Fraser,  Project  Manager 

5.  Research  Assistants 

* Summer  (May-August)  will  have  greater  average 
study  team  time  commitment  than  the  academic  year 


Benefits  on  Personnel 


28,446 


Study  Team  Travel  to  ARL  Libraries  and  Meetings 


19,000 
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[16  trips  @$1200  per  trip] 


National  Workshop  [at  end  of  Phase  II] 


1,250 


Communications  [supplies,  telephone,  copying,  etc.] 


7.250 


Total 


$199,990 


The  primary  expenses  will  be  for  study  team  personnel,  travel  expenses  to  conduct  field  tests  and  participate  in 
various  meetings,  and  various  communications  support.  The  study  would  be  conducted  as  a fixed-price  contract. 
One-fourth  of  the  project  costs  will  be  paid  upon  formal  agreement  to  conduct  the  study,  with  one- fourth  of  the 
project  costs  paid  at  the  conclusion  of  each  of  the  three  stages. 

Project  Communications 

The  study  team  anticipates  regular  and  ongoing  communication  with  participating  institutions  as  well  as  all 
ARL  libraries  via  the  ARL  New  Measures  Project  Initiative  Web  site  (on  the  ARL  or  another  Web  site).  ARL  staff 
as  well  as  possibly  other  staff  from  ARL  participating  libraries  will  have  responsibility  to  mount  project  information, 
updates,  project  reports,  issue  papers,  and  other  items  on  that  Web  site. 

In  addition,  a discussion  list  related  to  the  project  will  be  operated  and  maintained  by  ARL  for  regular  posting 
of  information  and  for  the  exchange  of  ideas  and  views  related  to  the  project  and  the  development  of  statistics  and 
performance  measures  related  to  the  networked  environment.  Regular  meetings  (either  in  conjunction  with  other 
professional  meetings  or  electronically)  will  occur  with  the  advisory  committee  to  discuss  project  activities.  The 
project  Advisory  Committee  Chairs  and  project  liaisons  (Miller  and  Schmidt)  will  keep  other  committees  at  ARL 
apprised  of  project  activities  as  needed. 

Use  of  Project  Information  and  Findings 

The  study  team  reserves  the  right  to  use  data  and  findings  from  this  project  in  other  future  studies  and  research 
efforts.  Indeed,  much  of  the  insight  and  information  that  the  study  team  brings  to  this  effort  is  a result  of  years  of 
previous  research  and  related  projects.  In  addition,  McClure,  Bertot,  and  Shim  reserve  the  right  to  publish  papers 
(hopefully  in  conjunction  with  project  participants)  and  otherwise  use  information  from  this  project  for  other 
educational  and  instructional  purposes.  Any  papers  submitted  for  publication  would  be  provided  to  the  liaison  for 
review  and  comment  prior  to  actual  publication  and  would  acknowledge  the  support  of  the  project.  Data  and  related 
findings  from  the  project  would  be  reported  in  aggregate  form  only  and  would  not  be  linked  to  individual  libraries. 

Importance  of  the  Project 

To  some  extent,  ARL  libraries  are  under-representing  and  under-counting  the  range  of  services  and  resource 
provision  in  which  they  are  engaged  because  there  are  no  agreed  upon  means  to  count  and  describe  service  provision 
in  the  networked  environment.  In  addition,  resource  allocation  decisions  are  further  complicated  by  a limited 
understanding  of  the  use  and  costs  of  services  provision  in  the  networked  environment.  Given  this  situation,  library 
administrators  are  oftentimes  unable  to  demonstrate  the  importance  and  impact  of  such  networked-based  services. 
Such  is  true  for  individual  libraries  as  well  as  for  ARL  libraries  as  a group. 

Clearly,  the  problems  and  issues  identified  in  this  prospectus  regarding  counts  and  measures  of  services  and 
resources  in  the  networked  environment  will  not  disappear  in  the  near  future.  Indeed,  these  issues  are  only  likely  to 
increase  in  importance  as  the  networked  environment  evolves  with  a range  of  new  services  and  resources.  Support 
for  this  project  provides  ARL  and  a group  of  ARL  libraries  to  take  a leadership  role  in  the  process  of  developing 
statistics  and  measures  for  services  and  use  in  the  networked  environment.  It  provides  a basis  for  ARL  libraries  to 
formally  begin  work  with  selected  database  vendors  to  reach  agreement  on  possible  data  reporting  activities.  And... 
the  project  will  begin  to  develop  linkages  between  these  statistics  and  performance  measures  and  a range  of  higher 
education  outcomes. 
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Most  ARL  libraries  need  such  statistics  and  measures  now.  The  sooner  work  is  initiated  on  a project  such  as  that 
outlined  in  this  prospectus,  the  sooner  such  statistics  and  measures  can  be  used  to  support  resource  allocation 
decisions,  services  provision  and  assessment,  and  strategic  planning. 


Association  of  Research  Libraries 


March  22, 2000 

To:  Prospective  E-Usage  Project  Participants 

From:  Rush  Miller,  University  of  Pittsburgh 

Sherrie  Schmidt,  Arizona  State  University  Library 

E-Usage  Project  Co-chairs 

Re:  ARL  Project  on  Developing  Statistics  and  Performance  Measures  to  Describe 

Electronic  Information  Services  and  Resources  in  ARL  Libraries 

RESPONSE  REQUESTED  BY  APRIL  7,  2000 

Thank  you  for  participating  in  the  February  27-29  Scottsdale  E-Usage  project 
Planning  Meeting  by  sending  your  institutional  representative(s).  This  meeting 
facilitated  a rich  exchange  of  information  about  electronic  information  services  and 
served  to  define  the  parameters  of  an  ARL  project  in  this  area. 

As  indicated  at  the  meeting  at  Scottsdale,  you  are  now  invited  to  respond  to  a 
call  for  participation  in  this  project.  We  need  your  response  by  April  7 in  order  to 
initiate  project  activities  by  May  12.  A study  prospectus  describing  the  goals, 
methodology,  and  deliverables  is  enclosed.  The  prospectus  was  prepared  by  project 
consultant  Charles  R.  McClure  with  advice  and  feedback  from  the  co-chairs  and  ARL 
staff.  We  believe  this  study  prospectus  will  provide  enough  information  for  you  and 
your  key  executives  to  make  an  informed  judgement  about  participation. 

Each  participating  library  is  expected  to  contribute  $10,000  to  underwrite  the 
operating  costs  of  the  study.  In  addition,  participating  libraries  will  need  to  contribute 
some  staff  time  and  other  supporting  resources.  Actual  levels  of  participation  may  vary 
by  library  depending  on  a number  of  factors. 

We  are  looking  for  at  least  12  and  no  more  than  20  institutional  participants. 
These  institutions  should  have  an  active  electronic  information  services  program  and 
some  experience  with  statistics  and  performance  measures  for  electronic  resources.  The 
participating  libraries  should  also  reflect  a balanced  representation  of  ARL  member 
institutions  in  terms  of  type  (pub he /private),  size,  and  geographic  location. 

ARL  intends  to  finalize  the  contractual  arrangements  with  the  Florida  State 
University  Information  Management  Use  and  Policy  Institute  by  early  May  and  initiate 
the  project  activities  before  the  ARL  Membership  Meeting  in  May.  Ah  study  findings 
and  recommendations  will  be  provided  to  the  full  ARL  membership. 

Please  respond  directly  to  Martha  Kyrilhdou  martha@arl.org  as  to  whether  you 
would  like  to  participate  and  why  participation  is  important  to  your  institution.  If  you 
would  tike  additional  information,  please  contact  us,  the  project  co-chairs,  Sherrie 
Schmidt  <sherrie.schmidt@asu.edu>  and/or  Rush  Miller  <rgmiher+@pitt.edu>. 


21  Dupont  Circle  NW 
Washington,  DC  20036 
202  296  2296  telephone 
202  872  0884  fax 
http:/ /www.arl.org/ 
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EXECUTIVE  SUMMARY 

This  report  provides  a description  of  activities  and  findings  from  Phase  I of  the  Association  of 
Research  Libraries  (ARL)  sponsored  study,  ARL  E-Metrics  Project:  Developing  Statistics  and 
Performance  Measures  to  Describe  Electronic  Information  Services  and  Resources  for  ARL 
Libraries.  Overall,  Phase  I finds  that  a number  of  ARL  libraries  participating  in  the  project  have 
developed  useful  strategies  and  approaches  for  developing  statistics  and  performance  measures  to 
describe  use,  users,  and  uses  of  electronic  and  networked  information  services  and  resources. 
Despite  these  strategies  it  appears  to  be  too  early  to  offer  “best  practices”  in  developing  and  using 
such  statistics  and  performance  measures.  The  study  also  identified  a number  of  key  issues  that 
will  require  additional  attention  as  the  project  continues  into  Phase  II. 

This  study  began  in  April  2000.  The  three  primary  goals  of  this  project  are  to: 

• Develop,  test,  and  refine  selected  statistics  and  performance  measures  to  describe 
electronic  services  and  resources  in  ARL  libraries; 

• Engage  in  a collaborative  effort  with  selected  database  vendors  to  establish  an 
ongoing  means  to  produce  selected  descriptive  statistics  on  database  use,  users,  and 
services;  and, 

• Develop  a proposal  for  external  funding  to  maintain  the  development  and  refinement 
of  networked  statistics  and  performance  measures. 

A group  of  24  ARL  libraries  funded  the  study  and  are  participating  in  it;  this  project  is  under 
contract  with  Florida  State  University’s  Information  Use  Management  and  Policy  Institute  and  is 
directed  by  Charles  R.  McClure,  Wonsik  “Jeff’  Shim,  and  John  Carlo  Bertot  under  the  leadership 
of  project  co-chairs,  Sherrie  Schmidt,  Dean  of  University  Libraries,  Arizona  State  University 
Library,  and  Rush  Miller,  University  Librarian  and  Director. 

Phase  I was  Knowledge  Inventory  of  ARL  Libraries  and  Organizing  an  ARL  Working  Group 
on  Database  Vendor  Statistics  (May,  2000  - October,  2000).  The  two  objectives  of  this  phase 
were  to  (1)  identify  and  describe  the  current  state  of  the  art  of  statistics  and  performance  measures 
for  networked  services  and  resources  in  ARL  libraries,  and  (2)  organize  an  ARL  Working  Group 
on  Database  Vendor  Statistics  to  begin  discussions  with  database  vendors. 

The  study  built  upon  a conference  held  in  Scottsdale,  AZ  in  February  2000  that  was  intended 
to  organize  the  project.  Phase  I relied  on  the  following  types  of  data  collection  methods: 

• Survey  questionnaires; 

• Site  visits  to  selected  libraries; 

• Sample  vendor  reports  supplied  by  members  of  Vendor  Statistics  Working  Group; 

• Sample  library  generated  reports  obtained  from  project  participants;  and, 

• Follow-up  interviews  with  participants  as  necessary. 

These  efforts  produced  a number  of  findings  as  well  as  identified  key  issues  and 
recommendations  that  are  detailed  in  this  report.  It  is  important  to  stress  that  the  findings  and 
recommendations  are  based  on  data  from  participating  libraries  and  may  not  be  generalizable  to 
the  larger  group  of  ARL  libraries. 

Key  findings  from  the  study  include  the  following: 
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• Findings  from  the  survey : Analysis  of  the  E-Metrics  survey  responses  reveals  a wide  range 
of  data  collection  and  use  activities  among  project  participants.  It  appears  that  measures 
related  to  patron  accessible  resources  and  costs  are  collected  more  consistently  and 
systematically  than  measures  related  to  electronic  resource  use  or  users  of  those  resources. 
Due  to  the  often  inconsistent  and  non-comparable  nature  of  vendor  supplied  statistics, 
libraries  seem  to  have  considerable  difficulty  in  tracking  overall  electronic  database  usage 
and  use  patterns. 

The  collected  data  seem  to  be  shared  widely  among  library  staff  and  with  parent 
institutions.  However,  the  manner  in  which  the  information  is  communicated  and  the 
nature  of  the  reporting  process  appear  to  be  limited.  Data  are  most  often  used  to  make 
purchasing  decisions  for  licensed  vendor  materials.  People  also  indicated  various  uses  of 
the  data  for  the  purpose  of  internal  and  external  reporting  and  service  assessment  and 
evaluation. 

Regarding  the  most  important  issues  related  to  performance  measurement  of  networked 
resources  and  services,  the  majority  of  respondents  cite  the  lack  of  consistent  and 
comparable  statistics  from  database  vendors  as  the  most  serious  problem.  Relatively  few 
respondents  recognized  or  identified  problems  associated  with  the  library's  inability  to 
process  and  utilize  collected  data. 

• Findings  from  vendor  reports : Analysis  of  usage  statistics  from  12  major  database  vendors 
reveal  that  there  is  a wide  range  of  different  practices  and  that  progress  should  be  made  in 
several  areas  including  standardization  of  core  statistics,  report  delivery  method,  and  assuring 
the  provision  of  definitions  of  reported  statistics.  There  are  some  signs  in  the  way  vendors 
report  data  that  indicate  increased  cooperation  between  libraries  and  the  vendors. 

• Findings  from  site  visits  (VT,  U Penn.,  Yale,  NYPL):  Libraries  reside  under  different 
operating  environments  and  have  very  different  needs  in  terms  of  data  to  describe  electronic 
services  and  resources.  The  environment  differs  because  of  the  institution's  involvement  with 
the  library  operation,  the  library's  top  management  attitude  toward  evaluation  efforts,  and  the 
library’s  needs  related  to  data. 

Libraries  have  a serious  problem  managing  information  describing  the  use  of  electronic 
resources  and  services.  This  is  particularly  the  case  with  regard  to  licensed  vendor 
materials  primarily  because  descriptive  data  often  reside  under  vendor  control.  Libraries 
often  have  to  manage  different  interfaces  to  obtain  different  types  of  resources,  and, 
accordingly,  usage  statistics  are  distributed  among  typically  several  dozen  database 
vendors  and  consortia.  Due  to  a lack  of  standardized  reporting  practices,  usage  reports  are 
difficult  to  consolidate,  or  it  takes  an  enormous  amount  of  effort  to  collect  such  data. 
Non-vendor  based  data  collection  efforts  to  describe  electronic  services  and  resources 
appear  to  have  received  less  attention  than  database  efforts. 

While  libraries  are  making  progress  in  some  areas  of  measurement  of  electronic  resources, 
libraries  have  yet  to  succeed  in  producing  a coherent  plan  or  strategy  for  using  and  reporting 
statistics  and  measures  related  to  electronic  information. 

In  addition,  the  study  identified  a number  of  issues  that  will  require  additional  discussion  and 
resolution: 
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• Complexity  of  the  topic : participating  libraries,  vendors,  the  study  team,  and  users 
may  not  understand  the  complexity  of  developing  statistics  and  performance 
measures  for  electronic  services  and  resources. 

• Diverse  context  for  developing  statistics  and  performance  measures : each  of  the 
various  ARL  libraries  operates  in  a unique  setting  that  affects  the  development  and 
use  of  specific  statistics  and  measures. 

• ARL  library  responsibilities  and  level  of  effort : there  are  a range  of  internal  factors 
that  affect  the  degree  to  which  the  library  can  provide  resources  and  an  adequate 
level  of  effort  to  collect  data  needed  for  such  statistics  and  performance  measures. 

• Focus  on  non-vendor  based  data  sources : there  are  a number  of  statistics  and 
measures  that  may  be  developed  that  do  not  depend  on  the  database  vendors  and 
libraries. 

• Coordination  among  libraries  and  library  organizations : there  are  numerous 
libraries,  organizations  such  as  NISO,  NCLIS,  ICOLC,  etc.  who  are  interested  in 
developing  standards  for  electronic  and  networked  services  and  resources  whose 
efforts  will  need  some  coordination. 

The  report  discusses  these  issues  in  greater  detail  and  will  be  important  areas  for  attention  in 
Phase  II  of  the  study. 

Although  findings  from  Phase  I of  the  study  did  not  identify  a set  of  “best  practices’’  for 
developing  electronic  and  networked  statistics  and  performance  measures,  the  study  team  can 
recommend  a number  of  very  specific  strategies  that  can  assist  participating  libraries  better 
prepare  for  data  collection  to  produce  such  statistics.  These  strategies  include  developing  a 
culture  of  evaluation;  stressing  the  use  and  development  of  statistics  and  measures  in  strategic 
planning  documents;  reorganizing  for  assessment,  data  collection,  and  reporting;  and,  developing 
a data  advocate  within  the  library.  Part  4 of  the  report  details  these  and  other  strategies. 

The  Phase  I report  concludes  with  a discussion  of  next  steps  to  be  taken  in  Phase  II.  These 
next  steps  are  based  on  the  activities  and  objectives  as  outlined  in  the  original  study  proposal. 
Nonetheless,  these  steps  include: 

• Developing  and  field-testing  possible  statistics  and  performance  measures  to  describe 
services  and  resources  in  the  electronic  environment; 

• Addressing  the  key  issues  outlined  above  and  detailed  in  Part  4 of  the  Phase  I report; 

• Organizing  the  Vendor  Statistics  Working  Group  and  meeting  with  selected  vendors  to 
discuss  statistics  and  measures  for  databases;  and, 

• Conducting  or  participating  in  a number  of  meetings  during  Phase  II  to  coordinate  the  library 
community’s  efforts  to  develop  such  statistics  and  measures. 

Phase  II  will  be  completed  in  June  200 1 and  will  result  in  a short  manual  that  proposes  statistics 
and  measures  that  libraries  can  use  to  describe  and  assess  electronic  services  and  resources. 
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PHASE  ONE  SUMMARY 

During  the  ARL  E-Metrics  Project's  Phase  One,  the  study  team  successfully  completed  data 
collection  and  analysis  along  a broad  range  of  activities  and  issues  related  to  measures  of 
networked  information  resources  and  services.  The  results  indicate  that  many  libraries  are  in  the 
initial  phases  of  collecting  data  on  electronic  resource  use  and  that  libraries  need  to  develop 
systematic  approaches  to  the  measurement  of  overall  library  performance.  The  study  also  shows 
that  libraries  are  experiencing  serious  information  management  problems  due  to  inconsistent  and 
non-comparable  statistics  supplied  by  database  vendors.  Concerted  effort  to  address  this  problem 
is  identified  as  a top  priority  during  the  Phase  Two  of  the  E-Metrics  project. 

1.  About  the  ARL  E-Metrics  Project 

The  ARL  E-Metrics  Project  is  one  component  of  the  ARL  New  Measures  Initiative,  an  effort 
that  tries  to  respond  to  the  needs  for  (1)  demonstrating  the  library’s  impact  in  areas  important  to 
an  institution  and  its  stakeholders,  and  (2)  maximizing  the  effective  use  of  resources  by 
benchmarking  best  practices  in  order  to  save  or  reallocate  resources. 

The  ARL  E-Metrics  Project  supports  an  investigation  of  measurement  of  library  performance 
in  the  networked  information  environment.  ARL  has  had  limited  success  in  collecting  data  on 
expenditures  for  electronic  resources  over  the  last  six  years  but  a more  comprehensive 
investigation  is  made  possible  through  this  project.  The  24  ARL  libraries  listed  in  Exhibit  1 are 
contributing  financial  and  staff  resources  (listed  in  Exhibit  2)  for  this  exciting  undertaking.  This 
project  is  under  contract  with  Florida  State  University’s  Information  Use  Management  and  Policy 
Institute  and  is  directed  by  Charles  R.  McClure,  Wonsik  “Jeff’  Shim,  and  John  Carlo  Bertot 
under  the  leadership  of  project  co-chairs,  Sherrie  Schmidt,  Dean  of  University  Libraries,  Arizona 
State  University  Library,  and  Rush  Miller,  University  Librarian  and  Director. 

Information  on  this  project  and  related  activities  is  located  at: 

ARL  E-Metrics  Project:  http://www.arl.org/stats/newmeas/e-metrics.html 

ARL  New  Measures  Initiative:  http://www.arl.org/stats/newmeas/newmeas.html 

ARL  Expenditures  for  Electronic  Resources:  http://www.arl.Org/stats/arlstat/#sup 


Exhibit  1.  ARL  E-Metrics  Participating  Institutions 


University  of  Alberta 

Arizona  State  University 

Auburn  University 

University  of  Chicago 

University  of  Connecticut 

Cornell  University 

University  of  Illinois-Chicago 

University  of  Manitoba 

University  of  Mary land-Col  lege  Park 

University  of  Massachusetts 

University  of  Nebraska-Lincoln 

University  of  Notre  Dame 

University  of  Pennsylvania 

Pennsylvania  State  University 

University  of  Pittsburgh 

Purdue  University 

University  of  Southern  California 

Texas  A&M  University 

Virginia  Polytechnic  Institute  and  State  University  (Virginia  Tech) 

University  of  Western  Ontario 

University  of  Wisconsin  Madison 

Yale  University 

Library  of  Congress 

The  New  York  Public  Library,  the  Research  Libraries 
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Exhibit  2.  ARL  E-Metrics  Project  Participants 


ARL  Library 

University  of  Alberta 

Arizona  State  University 

Auburn  University 

University  of  Chicago 

University  of  Connecticut 

Cornell  University 

University  of  Illinois-Chicago 

University  of  Manitoba 

University  of  Maryland-College  Park 

University  of  Massachusetts 

University  of  Nebraska-Lincoln 

University  of  Notre  Dame 

University  of  Pennsylvania 

Pennsylvania  State  University 

University  of  Pittsburgh 

Purdue  University 

University  of  Southern  California 

Texas  A&M  University 

Virginia  Tech 

University  of  Western  Ontario 
University  of  Wisconsin  Madison 
Yale  University 
Library  of  Congress 
The  New  York  Public  Library 
Association  of  Research  Libraries 
* Director  or  Dean  of  Libraries 


Contacts 

Ernie  Ingles*,  Karen  Adams,  Kit  Wilson,  and  Ernie  Howe 

Sherrie  Schmidt*  and  Kurt  Murphy 

Stella  Bentley*  and  Glen  Anderson 

Martin  Runkle*  and  James  Mouw 

Brinley  Franklin* 

Sarah  E.  Thomas*  and  Christian  Boissonnas 

Sharon  A.  Hogan*  and  Deborah  Blecic 

Carolynne  Presser*  and  Susan  Miller 

Charles  B.  Lowry*,  Irma  F.  Dillon,  and  Betty  Day 

Margo  Crist*  and  Gordon  Fretwell 

Joan  Giesecke*  and  Beth  McNeil 

Jennifer  A.  Younger*  and  Jo  Bessler 

Paul  H.  Mosher*,  Joe  Zucca,  and  Michael  Winkler 

Nancy  L.  Eaton*,  Kimlyn  Patishnock,  and  Sally  Kalin 

Rush  G.  Miller*,  Pamela  Vance,  and  Fern  Brody 

Emily  R.  Mobley*  and  Nancy  Hewison 

Jerry  D.  Campbell*  and  Joyce  Toscan 

Fred  M.  Heath* 

Eileen  Hitchingham*  and  Paul  Metz 

Joyce  C.  Garnett*  and  Lorraine  Busby 

Kenneth  Frazier*  and  Sandra  Guthrie 

Scott  Bennett*,  Ann  Okerson,  and  Kimberly  Parker 

Dianne  Kresh*  and  Barbara  Morland 

William  Walker*  and  Ann  Thornton 

Duane  Webster*  and  Martha  Kyrillidou 


2.  Organization  of  the  Report 

This  report  summarizes  the  activities  and  findings  during  the  first  phase  and  is  organized  as 
follows: 


Part  1:  Introduction  to  the  E-Metrics  Project 

Section  1 provides  a general  discussion  of  problems  and  project  goals. 

Section  2 discusses  related  projects  and  studies  that  address  the  same  or  similar  problems. 
Section  3 is  devoted  to  description  of  the  Phase  One  objectives  and  goals. 

Part  2:  Phase  One  Methodology 

Provides  information  concerning  data  collection  methods,  logistics  and  data  analysis. 

Part  3:  Summary  of  Findings 

Section  1 describes  the  results  from  the  E-Metrics  survey  conducted  during  August  2000. 
Section  2 provides  comparison  of  major  12  vendor  reports  in  terms  of  reported  statistics  and 
delivery  methods. 

Section  3 provides  the  findings  from  the  site  visits. 
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Part  4:  Conclusions  and  Next  Steps 

Provides  preliminary  suggestions  based  on  the  summary  findings  and  directions  for  the 
remainder  of  the  project. 


ERIC 
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PART  ONE:  INTRODUCTION 
1.  E-Metrics  Project  Overview 

Research  libraries  are  increasingly  concerned  about  the  difficulty  they  face  in  describing  their 
performance  in  providing  electronic  information  services  and  resources.  This  concern  grows  from 
lack  of  reliable  and  consistent  statistics  that  can  describe  how  successful  libraries  are  in  fulfilling 
user  needs  in  the  fast  paced  network  environment.  Several  factors  that  contribute  to  the 
seriousness  of  the  problem  include: 

• Increased  use  of  electronic  resources.  Over  the  years  the  expenditure  of  electronic 
resources,  or  at  least  the  proportion  of  electronic  resources  to  the  overall  materials  budget, 
has  increased  significantly.  According  to  the  ARL  Supplementary  Statistics , 1997-98,  ARL 
libraries,  on  average,  spent  8.85%  of  their  materials  budget  on  purchasing  electronic 
resources  in  one  form  or  another.  Combined,  100  ARL  libraries  reported  spending  a total  of 
$49  million  on  electronic  resources.  Without  follow-up  statistics,  it  is  difficult  to  tell  what 
kind  of  trajectory  we  see  in  terms  of  growth  of  the  expenditure.  But  there  is  no  doubt  that  the 
proportion  has  increased  over  the  last  couple  of  years  and  will  grow  in  the  foreseeable  future. 
There  is  a substantial  frustration  echoed  in  the  library  community  for  not  being  able  to  have 
data  that  show  at  least  the  usage  of  these  resources.  At  a minimum,  ARL  libraries  need  a 
range  of  data  about  the  use  of  electronic  resources  to  justify  the  growing  expenditure. 

• Impact  on  traditional  resources  and  services.  Electronic  resources  and  delivery  of 
information  electronically,  most  commonly  over  the  World  Wide  Web  (WWW),  are  not 
simple,  independent  add-ons  to  the  existing  library  collections  and  services.  Libraries  want  to 
know  how  the  introduction  and  proliferation  of  electronic  materials  affect  the  use  of  existing 
print  based  collection,  overall  role  of  the  library,  user  behaviors,  and  the  value  that  libraries 
can  offer  to  the  user  community  including  supporting  institutions.  Without  specific  data,  it  is 
not  possible  to  make  informed  decisions  that  can  adequately  respond  to  both  internal  and 
external  needs. 

• Enabling  peer  comparison.  One  of  the  important  aspects  of  collecting  data  is  to  be  able  to 
make  reasonable  comparison  among  peer  institutions.  It  is  not  possible  to  accomplish  such 
comparisons  without  agreeing  on  a set  of  measures  and  definitions  to  be  used  across  the 
institutions.  Benchmarking  is  not  limited  only  to  the  common  statistics  and  measures  we  try 
to  collect.  There  is  a need  to  look  at  the  processes  and  procedures  that  other  libraries  adopt  to 
deal  with  the  problems  in  the  hope  that  libraries  can  learn  something  useful  that  can  benefit 
themselves. 

• Common  data  to  support  goals  and  objectives.  If  research  libraries  want  to  position 
themselves  as  effective  knowledge  managers  in  the  digital  age  and  as  providers  of 
educational  value  and  outcome,  they  should  have  data  that  can  support  their  mission,  goals, 
and  objectives. 

Together,  these  factors  form  a basis  for  the  development  of  standardized  network  statistics 
and  performance  measures. 

The  first  goal  of  the  E-Metrics  project  has  three  objectives:  (1)  To  address  the  problem  of 
developing  a set  of  statistics  and  measures  that  can  better  describe  electronic  resources  and 
services  for  ARL;  2)  To  engage  in  a collaborative  effort  with  selected  database  vendors  to 
establish  an  ongoing  means  to  produce  useful  statistics  on  database  use,  users  and  services;  and  3) 
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To  develop  a proposal  for  external  funding  to  maintain  the  development  and  refinement  of 
networked  statistics  and  performance  measures.  More  specifically,  the  study  investigates  answers 
to  the  following  questions: 

• What  existing  techniques  and  approaches  are  ARL  libraries  using  to  produce  statistics  and 
performance  measures  to  describe  networked  information  services  and  resources?  What  can 
be  learned  from  these  techniques  that  could  be  generalized  to  other  libraries? 

• For  what  purposes  and  for  what  audiences  are  networked  statistics  and  measures  necessary? 

• Which  types  of  networked  services  and  resources  should  be  described,  how  should  they  be 
defined  and  operationalized,  and  how  should  the  data  be  collected  to  insure  reliable  and  valid 
data? 

• What  performance  or  quality  indicators  are  necessary  to  describe  the  impact  and  success  of 
such  networked  services? 

• How  might  such  statistics  and  measures  be  best  reported? 

• What  linkages  are  possible  to  establish  between  these  statistics  and  performance  measures 
with  selected  outcomes  from  higher  education? 

By  bringing  a critical  mass  of  ARL  libraries  to  the  problem  solving  process,  it  is  possible  to 
bring  a unified  approach  to  the  problem  for  maximum  impact.  An  important  aspect  of  the  project 
is  to  foster  an  environment  in  which  libraries  work  together  toward  a common  goal  as  they  have 
done  successfully  in  many  other  areas-it  is  not  possible  to  resolve,  by  individual  effort,  the  issues 
and  challenges  of  network  services  and  resources  measurement  ARL  libraries  face  today. 

The  project  adopted  a three-phased  approach  to  answer  the  research  questions  identified 
above. 

(1)  Initial  Phase  (May-October  2000):  Take  inventory  of  current  practices  at  ARL  libraries 
as  to  statistics,  measures,  processes,  and  activities  that  pertain  to  networked  resources  and 
services. 

(2)  Second  Phase  (November  2000-June  2001):  Based  on  the  knowledge  inventory  and 
drawing  from  previous  initiatives  such  as  ICOLC  (International  Coalition  of  Library  Consortia), 
identify  and  field  test  an  initial  set  of  statistics  and  measures  to  assess  the  degree  to  which  such 
data  collection  is  possible  and  the  collected  data  are  comparable  among  member  libraries. 

(3)  Final  phase  (July  2001 -December  2001):  Proposal  of  a final  set  of  refined  measures  to 
ARL,  complete  with  data  descriptions  and  data  collection/analysis/use  guidelines. 


2.  Related  Projects  and  Studies 

Libraries  of  all  types  recognize  the  importance  of  developing  statistics  and  performance 
measures  to  describe  the  usage  and  uses  of  their  networked  resources  and  services.  This 
importance  is  evident  in  the  number  of  initiatives  - both  national  and  international  - that  are 
underway  to  assist  libraries  assess  their  networked  resources  and  services.  These  initiatives  take 
different  approaches,  focus  on  different  types  of  libraries,  and  work  within  various  operating 
environments,  but  all  focus  on  developing  library  electronic  statistics  and  performance  measures. 
These  efforts  include  (initiatives  with  the  [Note:  researcher  involvement]  indicate  that  members 
of  the  study  team  have  direct  involvement  in  the  research  project): 
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• International  Coalition  of  Library  Consortia  (ICOLC).  Since  the  mid-90s,  an 
international  coalition  of  libraries  - predominantly  academic  - have  been  working 
towards  a standard  set  of  definitions  for  online  database  services.  Current  definitions 
reflect  work  completed  in  November  1998  (see  http://www.librarv.vale.edu/ 
consortia/webstats.htmQ. 

• Equinox.  The  Equinox  project,  funded  by  the  European  Union,  primarily  focuses  on 
developing  performance  measures  for  networked  services  and  resources  in  European 
academic  libraries.  This  project  continues,  but  has  shifted  from  performance  indicators 
to  software  development  to  assist  libraries  assess  their  network-based  services  (see 
http://equinox.dcu.ie/). 

• International  Standards  Organization  (ISO).  The  ISO  continues  to  develop  its  library 
statistics,  of  which  network  statistics  are  a part  (see  ISO/CD  2789  Information  and 
Documentation  - International  Library  Statistics). 

• National  Information  Standards  Organization  (NISO).  NISO  is  undertaking  a review 
and  update  of  its  z39. 7 - Library  Statistics  standard.  This  review  and  update  will 
consider  network  services  and  resources  statistics  and  performance  measures.  As  of 
October  2000,  the  planning  committee  for  the  standard  is  just  forming  and  beginning  to 
meet  to  develop  the  review  process  (see  http://www.niso.org).  [Note:  researcher 
involvement]. 

• National  Commission  on  Libraries  and  Information  Science  (NCLIS).  NCLIS  continues 
its  work  in  standardizing  online  database  usage  statistics  and  reporting  mechanisms. 

This  project  largely  focuses  on  the  public  library  environment  (see 
http://www.nclis.gov).  [Note:  researcher  involvement]. 

• Institute  of  Museum  and  Library  Services  (IMLS).  IMLS  sponsored  the  researchers  to 
develop  national  network  statistics  and  performance  measures  for  public  libraries.  That 
project  resulted  in  a network  statistics  manual  for  public  libraries  (Bertot,  McClure,  and 
Ryan,  2000).  This  work  continues  with  renewed  sponsorship  by  IMLS  and  NCLIS  to 
develop  a national  data  collection  system  for  public  library  network  statistics  (see 
http://www.ii.fsu.edu).  [Note:  researcher  involvement]. 

• Council  on  Library  and  Information  Resources  (CLER).  CLIR  has  engaged  consultants 
to  review  the  state  of  the  art  of  network  statistics,  analysis,  and  presentation  in  the 
academic  library  and  consortia  environments.  The  report  is  expected  in  fall  2000,  and 
there  is  an  indication  that  this  work  will  continue. 

All  of  the  above  initiatives,  groups,  and/or  research  projects  focus  in  some  way  on  the 
development,  analysis,  and  presentation  of  library  network  statistics  and  performance  measures 
that  reflect  library  needs  to  describe  the  use  of  the  network  resources  and  services. 

A main  issue,  however,  is  the  extent  to  which  the  initiatives,  groups,  and/or  projects 
coordinate  with  one  another.  While  the  authors  have  direct  involvement  in  a number  of  the 
above-mentioned  initiatives  thus  providing  some  cross-fertilization,  there  is  a substantial 
tendency  for  the  projects  to  work  independently  of  one  another.  This  is  problematic  at  best, 
particularly  in  the  area  of  online  database  vendor  usage  statistics.  For  a host  of  reasons  including 
vendor  cooperation,  library  reporting  requirements,  and  library  management  needs  (there  are 
others  as  well,  but  these  are  critical),  more  coordination  and  cooperation  is  necessary  throughout 
these  projects.  It  is  imperative  to  avoid  the  “not  invented  here”  syndrome  and  to  work  in  unison 
as  much  as  possible  create  a standardized  approach  to  definitions  and  reporting  mechanisms. 
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To  the  extent  possible,  the  researchers  will  coordinate  with  the  other  groups.  This 
cooperation  is  dependent  on  a number  of  factors,  including  project  team  resources,  level  of  effort, 
and  the  willingness  of  the  other  parties  to  engage  in  tandem  efforts. 


3.  PHASE  One  objectives  and  Research  Questions 


PHASE  I of  the  E-Metrics  project  had  two  objectives:  1)  To  identify  and  describe  the  current 
state  of  the  art  of  statistics  and  performance  measures  for  networked  services  and  resources  in 
ARL  libraries,  and  2)  To  organize  an  ARL  Working  Group  on  Database  Vendor  Statistics  to 
begin  discussions  with  database  vendors. 


There  are  at  least  two  reasons  why  the  study  team  believes  that  a knowledge  inventory  of 
current  practices  is  the  logical  starting  point  for  the  E-Metrics  project: 

(1)  While  it  is  obvious  that  ARL  libraries  are  struggling  to  obtain  data  describing  electronic 
services,  the  project  should  not  disregard  library  accomplishments  in  this  area.  Rather,  the  study 
team  intends  to  place  the  project  in  continuum  of  those  efforts.  The  ARL-sponsored  meeting  on 
New  Measures  held  in  February  2000  in  Scottsdale,  Arizona  clearly  demonstrated  that  several 
libraries  are  taking  an  initiative  in  collecting  and  reporting  data.  The  study  team  intends  to 
investigate  and  highlight  some  of  the  current  practices  that  may  prove  useful  to  other  libraries. 


(2)  For  the  formal  adoption  of  statistics  and  measures,  there  is  a need  to  identify  potential 
barriers,  problems  and  issues  necessary  to  resolve  in  producing  and  using  statistics  and 
performance  measures  As  such,  PHASE  I is  an  "active  listening  stage"  where  we  try  to 
understand  the  challenges  and  problems  associated  with  the  tasks. 


The  research  questions  during  the  PHASE  I included: 


• What  kinds  of  statistics  and  measures  are  ARL  libraries  collecting  to  describe  electronic 
resources  and  services? 

• What  are  the  strategies  and  processes  adopted  by  ARL  libraries  to  collect  and  process  the 
data? 

• For  what  purposes  are  statistics  and  measure  being  collected? 

• What  is  the  extent  to  which  ARL  libraries  use  the  data  that  they  collect? 

• What  are  the  problems,  challenges,  and  barriers  associated  with  collecting,  processing 
and  using  network  statistics  and  measures? 

• How  can  we  identify  best  practices  among  ARL  libraries  in  regards  to  developing 
measures  and  procedures  to  describe  electronic  resources  and  services? 

Part  Two  describes  the  research  methodology  adopted  by  the  study  team  to  answer  the  above 
questions. 
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PART  TWO:  METHODOLOGY 

To  adequately  address  the  research  questions,  the  study  team  used  the  following  types  of  data 
collection  methods: 

• Survey  questionnaires; 

• Site  visits  to  select  libraries; 

• Sample  vendor  reports  supplied  by  members  of  Vendor  Statistics  Working  Group; 

• Sample  library  generated  reports  obtained  from  project  participants;  and 

• Follow-up  phone/email  interviews. 

1.  Survey  Questionnaires 

The  study  team  conducted  two  surveys  during  Phase  I that  enabled  the  research  team  to 
collect  a range  of  responses  to  a same  set  of  questions  and  thus  allow  for  comparisons  across  the 
respondents.  The  study  team  analyzed  the  surveys  to  compile  an  inventory  of  various  statistics 
and  measures  collected  by  participating  libraries.  The  responses  also  provide  primary  evidence 
with  which  to  identify  libraries  for  site  visits  conducted  after  the  surveys. 

Scottsdale  Survey 
(See  Appendix  A) 

The  survey  was  prepared  and  sent  out  to  the  Scottsdale  meeting  participants  by  Martha 
Kyrillidou  and  Julia  Blixrud  and  responses  were  compiled  by  the  ARL  Statistics  and 
Measurement  Program  staff..  The  data  were  provided  to  McClure  and  Shim  by  ARL  for  the 
preparation  of  the  meeting.  Shim  made  a preliminary  summary  presentation  during  the  meeting 
on  the  main  themes  in  the  responses 

(http://www.arl.org/stats/newmeas/scottsdale/scottsdalehtm/sldQQlhtm). 

Of  the  35  ARL  institutions  represented  at  the  meeting,  21  libraries  responded  to  the  survey. 
Also  of  35  libraries  represented,  24  institutions  have  decided  to  participate  in  the  current  project. 
Among  those  21  libraries  that  responded  to  the  survey,  15  libraries  are  now  participants  in  the 
project. 

The  survey  asked  the  following  four  questions. 

1.  What  do  you  need  to  know  to  make  good  resource  decisions  related  to  the  provision  of 
electronic  services  or  purchase  of  networked  resources?  What  information  do  you  need  to 
communicate  to  decision-makers  about  electronic  resources? 

2.  What  data  do  you  need  to  measure  the  impact  of  electronic  services  and  resources?  Do 
you  have  this  data  available  now? 

3.  What  are  you  currently  doing  at  your  institution  in  terms  of  measuring  electronic  services 
and  resources?  Describe  your  current  role  in  developing  measures  for  assessing  electronic 
resources  and  services. 

4.  What  are  your  primary  goals  in  participating  in  this  project  planning  session? 

Responses  to  questions  2 and  3 provide  us  with  some  indication  of  data  collection  activities  at 
responding  libraries.  Responses  to  the  first  question  also  show  the  level  of  sophistication  or 
articulation  on  the  issue  that  the  respondents  or  the  institutions  have. 
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Since  the  Scottsdale  survey  was  not  part  of  the  actual  E-Metrics  project,  this  report  does  not 
present  findings  from  that  survey.  The  data,  however,  contain  some  very  important  information 
that  may  prove  useful  for  the  E-Metrics  project  and  for  others  in  the  future.  Therefore,  we  provide 
a somewhat  detailed  summary  of  the  responses  in  Appendix  A. 

E-Metrics  Survey 
(See  Appendix  B) 

The  survey's  purpose  was  to  gain  focused  data  in  the  following  categories: 

• Currently  collected  measures; 

• Data  use  in  terms  of  specific  decision  making; 

• Issues  and  challenges  facing  libraries  in  the  data  collection  efforts;  and 

• Areas  in  which  libraries  have  best  practice  related  to  network  measurement  activities. 

The  survey  also  asked  the  libraries  to  include  sample  reports  pertaining  to  electronic 
resources  and  services-Note  that  the  second  and  third  categories  were  asked  as  open-ended 
questions. 

More  specifically,  the  survey  asked  the  respondents  to  list  measures  that  they  currently 
collect  in  the  following  five  categories: 

• Measures  related  with  patron  accessible  resources; 

• Measures  related  with  patron  use; 

• Measures  related  with  users; 

• Measures  related  with  cost;  and 

• Other  measures  that  cannot  be  classified  above  but  important  to  electronic  resources  and 

services. 

The  survey  also  asked  the  respondents  to  specify  for  how  long  each  measure  has  been  collected, 
how  frequently,  and  the  name  of  the  entity  if  the  data  are  requested  by  someone  or  an  entity  both 
inside  and  outside  the  library. 

The  study  team  circulated  a draft  version  of  the  survey  among  the  participants  of  the  ARL 
Working  Group  (WG)  on  Database  Vendor  Statistics  meeting  in  Chicago  on  July  7,  2000.  A 
revised  pre-test  version  of  the  survey  was  sent  to  4 libraries  that  agreed  to  take  part  in  the  testing. 
Based  on  the  comments  from  the  pre-test,  the  study  team  distributed  the  final  version  to  the 
project  participants  on  July  31,  2000  in  an  email  attachment.  Of  the  24  e-mailed  surveys,  the 
study  team  received  22  completed  surveys,  primarily  via  e-mail. 

Survey  responses  were  aggregated  in  an  Excel  spreadsheet  for  analysis.  For  measures 
collected,  two  research  assistants  read  the  responses  and  constructed  an  authority  list  for  each 
category  where  applicable.  They  then  went  back  to  each  response  to  tally  it  up  according  to  the 
authority  list.  The  analysis  and  summary  of  the  frequency  of  data  collection,  the  time  data 
collection  began,  and  the  person  or  organization  who  requested  the  data  proved  difficult  due  to 
the  variation  in  responses.  As  such,  the  study  team  created  simple  frequency  tables  for  these  types 
of  information. 
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Through  an  iterative  process,  the  study  team  developed  a data-derived  taxonomy  in  an  effort 
to  put  responses  to  open  ended  questions  (decision-making  and  issues)  into  some  context. 
Responses  were  then  assigned  to  specific  classes  in  the  taxonomy. 


2.  Site  Visits 

The  study  team  used  the  following  approach  to  selecting  the  best  practice  sites: 

• Reviewed  the  list  of  respondents  who  expressed  interest  in  the  benchmarking  process; 

• Reviewed  the  documents  provided  by  those  libraries  seeking  best  practices; 

• Considered  factors  such  as  the  type  of  library  (public  v.  private),  location,  and  unique 

situations  that  the  candidate  library  is  expected  to  bring  to  the  mix;  and 

• The  sample  for  the  site  visits  was  limited  to  the  active  participants  of  the  project. 

Readers  should  note  that  the  best  practices  that  are  identified  and  documented  in  this  report 
are  snapshots  of  a point  in  time.  As  libraries  develop  better  ways  to  deal  with  the  network 
measurement  issues,  today’s  best  practices  may  well  become  less  than  best  or  average  practices. 
Indeed,  one  of  the  goals  of  benchmarking  is  to  push  the  limits  of  current  practices  and  to 
continually  challenge  organizations  to  perform  more  effectively  and  efficiently. 

The  study  team  used  the  following  criteria  to  identify  candidates  for  site  visits. 

• Willingness  to  participate  as  indicated  in  the  E-Metrics  survey; 

• Evidence  that  the  library  engages  in  some  level  of  data  collection;  and 

• Knowledgeable  staff  with  whom  the  study  team  could  meet. 

Based  on  the  above  criteria,  the  study  team  identified  the  following  libraries  as  site  visit 
candidates: 

• Virginia  Tech  (VPI  & SU)  Library; 

• University  of  Pennsylvania  Library; 

• Yale  University  Library;  and 

• The  Research  Libraries  at  the  New  York  Public  Library. 

The  study  team  sent  invitations  to  the  candidate  libraries  early  August  so  that  the  visits  could 
take  place  no  later  than  the  end  of  August  to  avoid  any  problems  with  libraries  preparing  for  the 
fall  semester.  All  the  identified  libraries  accepted  the  invitations 

Exhibit  3 presents  the  list  of  questions  mailed  to  the  institutions  in  advance  of  the  site  visit  to 
facilitate  discussions  between  the  study  team  and  site  visit  library  staff. 

The  site  visits  occurred  during  the  week  of  August  21, 2000.  Shim  conducted  all  site  visits  as 
outlined  in  the  study  proposal.  Each  visit  lasted  on  average  about  4 hours  and  consisted  of  2-3 
sessions  of  group  discussions  centered  on  the  questions  in  Exhibit  3.  During  the  visits,  Shim 
collected  relevant  materials  such  as  annual  reports  and  library  internal  documents. 
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Exhibit  3.  Site  Visit  Discussion  Questions 


Questions  to  library  administrator 

1.  Why  are  you  collecting  information  about  electronic  (digital)  resources  and  services? 

2.  What  kinds  of  data,  measures,  or  statistics  will  best  serve  your  needs? 

3.  How  important  or  how  critical  is  it  to  have  that  information?  What  kinds  of  additional 
resources  you  are  willing  to  spend  to  obtain  that  information? 

4.  What  is  the  organizational  structure  and  procedures  for  collecting  and  reporting  data,  measures, 
and  statistics  related  to  networked  services  and  resources? 

5.  What  are  some  of  the  biggest  challenges  in  producing  and  using  the  information?  Tell  us  how 
your  library  solved  or  tried  to  resolve  one  of  the  challenges. 

6.  How  do  you  compare  data  collection  and  use  activities  related  to  digital  materials  with  those 
related  to  print  materials? 

7.  Do  you  see  any  changes  in  the  administration's  view  of  the  library's  role  as  the  library  increases 
the  portion  of  digital  materials  and  becomes  more  technology-oriented? 

8.  Please  tell  us  about  any  immediate  and  long-term  plans  related  to  measures  and  indicators  of 
electronic  services  and  resources. 

9.  Can  you  provide  us  with  any  written  reports  or  studies  related  to  producing  networked  data, 
statistics,  and  measures?  Are  there  any  reports  or  other  useful  information  on  your  website 
about  this  topic? 

Questions  to  librarians/staff  members 

1.  Can  you  describe  in  some  detail  what's  your  involvement  in  producing  statistics  and  measures 
relating  to  electronic  services  and  resources? 

2.  Could  you  discuss  your  approach  or  methodology?  What  tools,  technologies,  and  software  do 
you  use? 

3.  What  steps  do  you  take  to  try  to  ensure  the  collection  and  reporting  of  valid  and  reliable  data? 

4.  Can  you  describe  the  vendors  with  whom  you  work  to  obtain  data  and  reports  and  discuss  their 
level  of  interest  and  assistance?  Which  vendors  and  individuals  have  been  most  helpful  to 
you?  Which  have  been  least  helpful? 

5.  Can  you  describe  procedures  and  possible  interactions  with  vendors  that  might  assist  in  getting 
the  necessary  data  from  them? 

6.  What  are  the  biggest  challenges  or  problems  you  face  to  produce  the  measures  and  statistics? 
Please  tell  us  how  you  solved  or  tried  to  resolve  a specific  challenge. 


3.  Vendor  Reports 

During  the  July,  2000  ALA  annual  conference  in  Chicago,  the  ARL  Working  Group  (WG)  on 
Database  Vendor  Statistics  had  its  first  meeting.  The  WG  is  a smaller  set  of  study  participants 
(13)  that  focuses  on  the  statistics  describing  use,  users,  costs,  etc.,  of  vendor  databases.  During 
the  meeting,  members  of  the  Working  Group  were  asked  to  submit  a list  of  vendors  with  which 
they  spend  more  than  $20,000  for  annual  subscriptions  and  also  supply  sample  vendor  reports  and 
other  information  relevant  to  vendor  statistics. 

The  study  team  received  the  vendor  lists  from  nine  libraries.  Among  these  libraries,  some 
sent  us  the  top  ten  vendors,  while  others  included  a fairly  comprehensive  list  of  database  vendors 
with  whom  they  have  subscriptions.  The  study  team  reviewed  the  lists  and  identified  the  12  most 
frequently  listed.  Exhibit  4 lists  the  names  of  vendors  whose  usage  statistics  report  was  analyzed 
in  this  study. 
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Exhibit  4.  List  of  Compared  Vendor  Reports 


Academic  Press/IDEAL 

Bell  & Howell/ProQuest 

Ebsco 

Elsevier/ScienceDirect 

GaleNet 

HighWire 

ISI/Web  of  Science 

JSTOR 

Lexis-Nexis 

OCLC/FirstSearch 

Ovid 

Silver  Platter 

Given  the  fact  that  research  libraries  deal  with  several  dozens  of  database  vendors,  the  list  in 
Exhibit  4 may  seem  short.  However,  the  list  represents  a fair  portion  of  the  major  vendors  that 
have  business  with  the  academic  library  community.  Also,  it  is  not  feasible  to  summarize  the 
entire  variety  of  vendors  and  their  statistics. 

The  study  team  reviewed  the  sample  vendor  reports  sent  by  libraries  in  terms  of  the  statistics 
provided,  format  of  the  reports,  frequency  of  reporting,  statistics  access,  and  other  information 
that  can  provide  a concise  summary  of  vendor  offerings.  The  study  team  also  tried  to  match 
statistics  from  each  vendor  to  the  ICOLC  guidelines  as  libraries  and  vendors  both  seem  to  be 
leaning  toward  the  guideline  as  a current  yardstick  of  how  successful  a vendor  can  be  in 
delivering  usage  statistics.  The  ICOLC  guideline  is  attached  in  Appendix  C for  easy  reference. 

Finally,  the  study  team  reviewed  the  definitions,  descriptions,  or  explanations  of  the  usage 
statistics  provided  by  the  vendors  along  with  the  actual  reported  statistics.  This  provides  data 
regarding  how  much  agreement  there  is  among  library  vendors  as  to  what  the  reported  measures 
mean. 

One  caution  regarding  vendor  statistics  is  that  they  are  moving  targets.  At  the  time  of  this 
writing,  some  vendors  may  have  changed  some  aspect  of  their  reports  or  reporting  mechanism. 
Thus,  the  findings  presented  in  this  report  provide  a snapshot  at  a point  in  time,  and  serve  as 
providing  a standardized  review  of  today's  vendor  statistics. 

The  Washington  State  Library  compiled  a table  that  describes  database  vendor's  capability  (or 
plans)  in  terms  of  providing  the  ICOLC  compliant  statistics  for  the  Washington  Library 
Association  annual  conference,  May  17-19,  2000  in  Tacoma,  Washington.  The  document  is 
reproduced  in  Appendix  D.  The  listing  is  especially  useful  as  an  overview  of  vendor  statistics. 


4.  Limitations  of  the  Study 

The  study  sample  consists  of  a self-selected  group  of  24  libraries.  Thus,  there  is  substantial 
willingness  on  the  part  of  libraries  in  the  group  to  cooperate  in  the  data  collection  activities  partly 
because  of  the  vested  interest.  At  the  same  time  the  findings  in  this  report  may  not  represent  the 
ARL  member  libraries  as  a whole.  Rather,  as  it  will  be  explained  in  the  findings  section,  there 
are  unique  local  operating  environments  and  needs  that  each  ARL  library  faces  as  it  attempts  to 
measure  its  electronic  resources  and  services. 
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PART  THREE:  SUMMARY  OF  FINDINGS 

This  section  of  the  report  is  organized  by  the  findings  generated  from  the  E-Metrics  survey 
conducted  during  August  2000,  vendor  reports,  and  sites  visits. 

1.  E-Metrics  Survey  Findings 

The  findings  are  presented  in  the  order  and  form  of  the  original  survey.  The  intention  of  the 
survey  was  to  obtain  the  range  of  measures  currently  collected  among  participating  ARL  libraries 
rather  than  an  exhaustive  list.  Also,  the  survey  did  not  ask  the  libraries  to  supply  the  definitions 
of  the  measures  that  they  reported  on  the  survey  form.  As  such,  there  may  be  cases  in  which 
different  measures  mean  essentially  the  same  thing  or  vice  versa. 

Data  Collection 

1)  Measures  of  Patron  Accessible  Resources  (e.g.  # of  electronic  database  titles  served,  # of 
library  web  pages  in  service,  # of  e-books,  # of  full-text  e-joumals,  # of  librarians  providing 
electronic  reference.) 

Table  1.  Collected  Measures  on  Electronic  Resources  by  Number  of  Mentions 


Collected  Data 

No.  Mentioned 

Number  of  electronic  database  titles 

17 

Number  of  electronic  (full-text)  journals 

16 

Number  of  electronic  books 

6 

Number  of  library  web  pages  in  service 

4 

Number  of  new  electronic  titles 

3 

Number  of  cancelled  electronic  titles 

2 

Number  of  electronic  resources 

2 

Number  of  computer  files 

1 

Number  of  networked  CD-ROMs 

1 

Number  of  librarians  providing  electronic  reference 

1 

Number  of  public  workstations  available 

1 

Number  of  classes  on  electronic  resources 

1 

Other 

3 

As  illustrated  in  Table  1,  most  libraries  seem  to  keep  track  of  various  types  of  electronic 
materials.  In  terms  of  licensed  materials,  electronic  databases  and  electronic  full-text  journals 
represent  the  bulk  of  the  measures  related  to  electronic  collection.  There  were  two  libraries  that 
used  electronic  resources  to  denote  perhaps  electronic  databases.  The  same  applies  to  one  library 
that  collects  the  number  of  computer  files.  Most  libraries  that  collect  the  number  of  electronic 
databases  also  said  they  keep  separate  counts  for  full-text  journals.  Three  libraries  gather 
information  about  the  number  of  databases  added  or  deleted  periodically. 

Only  six  libraries  reported  that  they  collect  information  about  electronic  books  (e-books).  At 
this  point,  it  is  not  clear  whether  e-books  include  only  those  e-books  that  are  commercially 
available  through  companies  like  NetLibrary  or  scanned  copies  of  books  generated  locally. 

Four  libraries  report  that  they  collect  some  information  about  the  size  of  their  web  contents. 
However,  a page  in  one  library  can  be  quite  different  from  a page  in  another  library.  This 
information  can  provide  not  only  a trend-line  in  terms  of  the  amount  of  information  in  the  local 
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context  but  also  some  crude  measure  of  staff  productivity.  Among  the  less  frequently  mentioned 
measures  is  the  number  of  networked  CD-ROMs  (1).  This  may  be  interpreted  as  an  indicator  that 
CD-ROMs  are  gradually  disappearing  in  the  library  and  that  their  role  as  a major  networked 
resource  in  research  libraries  is  diminishing.  Also  only  one  library  reported  that  it  keeps  the 
public  workstation  count  available  in  the  library.  The  survey  did  not  produce  data  that  details  the 
number  of  public  workstations  available  in  ARL  libraries  over  the  years. 

The  measures  reported  in  the  'other"  category  include  the  number  of  digitized  images,  the 
number  of  print  or  CD-ROM  indexes  migrated  to  the  web  and  the  number  of  locally  mounted 
database  records.  All  three  measures  were  reported  from  the  same  library 

In  terms  of  the  collection  frequencies  of  these  measures,  libraries  provide  these  numbers 
annually  in  most.  While  unclear,  the  study  team  suspects  that  many  libraries  use  the  information 
for  reporting,  both  internal  and  external,  purposes.  Only  about  one-third  of  the  libraries  (8) 
indicated  that  they  collect  the  key  measures  monthly.  A few  libraries  said  they  obtain  the 
information  quarterly. 

There  is  a wide  variation  in  terms  of  when  the  data  collection  began  for  different  measures  in 
the  same  institution  as  well  as  for  the  same  measure  among  reporting  libraries.  For  instance,  one 
library  reported  that  it  started  collecting  the  number  of  electronic  databases  since  1991,  the 
number  of  electronic  journals  since  1995  and  the  number  of  web  pages  in  service  since  1998. 
Among  libraries  that  collect  the  number  of  electronic  databases,  9 out  of  17  libraries  indicated 
that  they  started  collecting  the  data  after  1997.  The  earliest  goes  back  to  1990  and  1991.  Three 
libraries  did  not  specify  when  they  began  data  collection. 

Seven  libraries  report  that  they  collect  the  data  to  fulfill  requirements  of  external 
organizations  such  as  ARL  and  CARL.  It  appears  they  are  referring  to  the  annual  ARL 
Supplemental  Statistics  that  collects  data  on  expenditures  for  electronic  resources 
(http://www.arl.Org/stats/arlstat/#sup).  Almost  an  equal  number  of  libraries  (6)  said  that  the  data 
were  required  by  library  administration  or  staff.  At  least  one  library  was  able  to  list  a rather 
comprehensive  group  of  consumers  of  the  collected  information:  University  and  Library 
Administration,  Library  Committee  responsible  for  networked  electronic  resources,  Serials 
Department,  Libraries  Electronic  Technologies  and  Services  (LETS),  bibliographers,  and  Library 
Unit  Heads.  Only  one  library  reported  that  the  information  is  required  as  part  of  strategic 
planning  efforts. 

2)  Patron  Use  Measures  (e.g.  # of  logins  or  visits,  # of  library  web  pages  accessed,  # of 
documents  downloaded,  # of  electronic  reference  service  transactions.) 

It  is  quite  difficult  to  summarize  the  responses  to  this  category  given  the  wide  range  of 
measures  reported  that  are  likely  to  be  confounded  with  seemingly  inconsistent  use  of  terms. 

As  far  as  the  vendor  materials  are  concerned,  nearly  all  libraries  responded  that  they  collect 
usage  statistics  as  provided  by  vendors.  These  include  sessions,  searches,  downloaded  records, 
rejected  logins  (that  is  tum-aways),  and  so  on.  Clearly,  it  is  not  possible  to  collect  these  measures 
uniformly  across  the  database  vendors  if  available  at  all. 

Several  libraries  seemed  to  have  a formal  report  that  summarizes  various  measures  available 
either  from  the  vendors  or  internally.  According  to  a sample  report  sent  by  one  library,  it 
compiles  a quarterly  ER  (Electronic  Resources)  Usage  Report  by  database,  including  measures 
such  as  sessions,  connect  time,  searches,  records  and  articles  (downloaded  or  viewed),  and  lock- 
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outs.  Obviously  there  are  a number  of  empty  rows  in  the  report.  Several  other  libraries  also  sent 
sample  reports  that  are  less  formal  but  nonetheless  attempt  to  fill  in  as  much  information  as 
possible  using  a common  set  of  core  measures  supplied  by  vendors. 

A handful  of  libraries  (4)  seemed  to  have  some  form  of  a click-through  mechanism  where 
they  capture  "attempted"  log-ons  to  electronic  databases  and  full-text  journals  from  their 
electronic  resource  pages.  One  library  goes  even  further  to  map  these  click-throughs  to  schools 
and  departments  based  on  IP  domains  associated  with  campus  units. 

Most  libraries  capture  information  about  library  webpage  usage  through  programs  such  as 
WebTrends  and  Analog.  However,  when  we  examined  the  sample  reports  from  select  libraries, 
they  are  mostly  raw  information,  such  as  page  access,  generated  by  the  software  programs.  The 
study  team  did  not  see  good  examples  of  succinct  summary  of  the  collected  information  other 
than  a few  factoids  such  as  top-ten  most  visited  pages  (or  areas)  and  peak  use  period. 

Other  use  measures  worth  mentioning  include  the  number  of  electronic  reference 
transactions,  the  number  of  documents  downloaded  from  electronic  reserve,  and  the  number  of 
electronic  document  delivery  requests  (e.g.,  from  Current  Contents  database).  About  half  of 
libraries  (10)  reported  that  they  collect  the  number  of  electronic  reference  transactions. 

In  terms  of  frequency  of  data  collection,  an  overwhelming  number  of  libraries  reported  that 
they  collect  statistics  monthly.  This  obviously  coincides  with  the  fact  that  many  database  vendors 
supply  monthly  usage  statistics  and  the  fact  that  many  library  system  log  files  (e.g.,  web  statistics) 
are  also  captured  on  a monthly  basis.  The  only  major  exception  is  the  number  of  electronic 
reference  transactions  with  which  4 out  of  10  libraries  that  report  the  measure  said  they  collect  it 
yearly  as  opposed  to  collecting  it  monthly. 

Again,  it  is  quite  recently  when  most  libraries  began  collecting  various  usage  statistics  such 
as  web  access  statistics,  licensed  database  use  statistics,  the  number  of  electronic  reference 
questions.  We  can  see  that  libraries  first  collected  input  oriented  information  such  as  database 
titles  and  gradually  moved  to  collecting  usage  statistics  as  they  received  information  from 
database  vendors  and  some  needs  (such  as  budget  justification)  required  them  to  have  these  types 
of  information. 

While  more  than  two  thirds  of  libraries  were  required  to  report  various  electronic  resource 
related  measures,  only  about  one  third  of  the  libraries  said  they  collect  usage  statistics  because 
someone  requested  them.  The  person  who  is  mostly  likely  to  request  data  is  either  collection 
development  manager  or  individual  collection  development  staff  members.  This  tells  us  that 
usage  statistics,  at  the  current  moment,  serve  primarily  for  collection  development  activities  such 
as  renewal/cancellation  of  subscription. 

3)  Measures  of  Users  of  Electronic  Resources  and  Services  (e.g.  % of  undergrads  who  have  used 
the  e-books,  % of  grads  who  have  used  the  electronic  reference  service,  # of  users  by  type  of 
services) 

Libraries  all  agree  that  they  do  not  have  a way  to  distinguish  individual  users.  This  is  a 
serious  problem  as  libraries  strive  to  collect  such  crucial  information  as  user  penetration  as  an 
indication  of  the  library  providing  value  to  the  user  community. 

With  the  introduction  of  OPACs,  libraries  were  able  to  track  who  was  borrowing  what 
materials  with  a reasonable  amount  of  effort.  Now  the  Web,  on  which  most  electronic  services 
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are  carried  out,  presents  a substantial  obstacle  as  far  as  libraries'  ability  to  know  about  their  users. 
Obtaining  this  information  is  not  impossible.  As  a matter  of  fact,  the  same  technology  that 
enabled  libraries  to  have  information  on-line  can  be  used  to  track  users  and  their  activities. 
However,  at  the  moment,  libraries  are  very  reluctant  to  put  any  obstacles,  such  as  forcing  users  to 
enter  library  card  number  and  a few  clicks  to  authenticate  themselves,  that  may  hinder  use  of 
electronic  materials.  Also,  this  has  to  do  with  the  fact  that  many  licensed  materials  are  essentially 
IP  validated  to  ease  the  burden  on  the  user  to  authenticate  themselves  to  each  database  and  also 
on  the  database  vendor  to  manage  access  with  too  much  overhead. 

The  only  exception  where  a comprehensive  set  of  data  are  available  is  two  libraries  tracking 
the  number  of  uses  originating  from  various  academic  units  including  libraries,  computer  labs  and 
other  buildings  identified  with  a set  of  IP  addresses.  In  rare  instances,  locally  mounted  databases 
can  keep  track  of  users  by  type  (faculty,  undergraduate,  graduate,  and  staff).  There  is  only  one 
mention  of  that  kind  of  setup. 

Otherwise,  libraries  collect  partial  information  of  bits  and  pieces  from  at  least  several  sources. 
Three  libraries  reported  that  they  ask  the  users  to  identify  themselves  (by  type)  when  they  submit 
electronic  reference  queries.  One  library  conducted  a user  survey  on  electronic  resource  use 
where  users  were  identified  by  type.  Another  library  said  they  monitor  user  comments  submitted 
electronically  as  to  what  type  of  users. 

In  two  instances  the  use  of  a proxy  server  for  obtaining  user  information  is  mentioned.  One 
library  has  information  about  users  by  remote  vs.  on-campus  connections.  The  other  library  has 
not  yet  collected  information  but  plans  to  use  the  proxy  server  log  to  obtain  user  demographics. 

4)  Cost  Measures  (e.g.  $ per  electronic  document  delivered,  cost  of  database  subscription  fee, 
expenditures  for  electronic  journals) 

Table  2 shows  the  frequency  of  cost  measures  reported  in  the  survey  responses.  Many 
libraries  collect  the  overall  cost  for  electronic  database  subscriptions.  All  libraries,  except  for  one 
library  with  no  response  for  this  category,  reported  that  they  have  this  information.  Some 
libraries  (8)  went  even  further  to  differentiate  the  costs  for  electronic  full-text  journals. 

Table  2.  Measures  of  Cost 


Collected  Data 

No.  Mentioned 

Total  costs  of  database  subscriptions 

21 

Expenditures  for  e-joumals 

8 

Cost  per  electronic  document  delivered 

4 

Cost  per  search 

3 

Average  cost  per  database  subscription 

2 

Cost  per  login 

2 

Other 

16 

No  response 

1 

Approximately  less  than  half  of  the  libraries  (9)  reported  that  they  link  the  cost  per  some  kind 
of  usage  measure  such  as  the  number  of  searches,  logins  or  documents  delivered.  In  some 
instances,  the  information  is  available  from  the  vendor  as  indicators  of  value  or  cost  savings. 
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Some  libraries  commented  that  calculating  cost  per  use  is  like  comparing  apples  and  oranges: 
they  are  incomparable.  While  this  may  be  true,  we  also  suspect  that  libraries  fear  that  producing 
the  cost  per  use  might  be  uncomfortable  if  the  ratio  turns  out  to  be  too  expensive. 

There  are  a variety  of  measures  collected  in  individual  libraries.  One  library  reported  a 
staggering  range  of  cost  measures  that  include  the  percentage  of  collections  budget  spent  on 
electronic  resources,  dollar  amounts  by  college,  cost  projections,  dollar  amounts  invested  in 
digital  library,  web  development,  electronic  resource  spending  per  student,  and  ratio  of 
paper/electronic  journals  costs.  All  these  measures  are  incorporated  in  their  library  strategic 
planning  document.  There  was  at  least  one  other  library  that  reported  projected  cost.  Some 
libraries  differentiated  one  time  costs  (such  as  JSTOR  membership)  and  on-going  subscription 
costs.  One  library  said  they  keep  the  cost  for  electronic  books  separately.  One  library  said  they 
collect  information  about  internal  and  external  sources  of  funding  for  electronic  resources. 

There  are  some  similarities  between  the  patron  accessible  resources  measures  and  cost 
measures  in  terms  of  how  frequently  libraries  collect  the  information,  when  they  began  collecting 
the  information,  and  who  asked  for  the  information.  Again,  except  for  a handful  of  cases,  many 
libraries  report  cost  information  annually,  perhaps  for  annual  budget  preparation.  Where  data  are 
reported  monthly  or  quarterly,  it  appears  that  the  libraries  have  a special  electronic  resource  cost 
report  type  of  arrangement  that  is  reviewed  by  a committee  in  charge  of  database  renewal  in 
addition  to  annual  budget  materials  where  these  numbers  are  used. 

It  seems  collecting  cost  measures  started  roughly  at  the  same  time  or  period  when  libraries 
started  counting  the  number  of  electronic  resources. 

Collection  of  cost  measures  at  6 libraries  was  required  as  part  of  the  ARL  Supplementary 
Statistics  survey  on  the  topic  ( http://www.arl.Org/stats/arlstat/# sup).  Nine  other  libraries  said  the 
information  was  requested  by  either  library  committee  or  management  (7)  or  outside  government 
body  (1)  or  for  strategic  planning  requirement.  One  library  said  that  they  reported  the  total  cost 
of  database  subscription  to  a magazine. 

5)  Other  Measures  related  with  electronic  resources  and  services  (such  as  service  quality,  the 
effects  of  library  use  on  research  and  instruction  or  the  percent  of  library  users  satisfied  with 
libraries  services) 

This  category  served  to  capture  any  on-going  data  collection  efforts  on  user  satisfaction, 
quality  and  value  of  services.  However,  it  appears  that  the  responses  were  not  limited  to 
electronic  services.  Instead,  what  emerges  from  the  responses  is  a list  of  tools,  shown  in  Table  3, 
libraries  use  to  measure  some  aspect  of  the  aforementioned  aspects  of  library  service. 

Table  3.  Sources  Used  to  Obtain  Information  Regarding  Service  Quality  and  User  Satisfaction. 


Instrument  Types 

No.  Mentioned 

User  satisfaction  survey 

7 

LibQUAL+ 

3 

Focus  Group 

2 

Library  Class  Evaluation 

2 

User  Panel  Feedback 

1 

User  Forum 

1 
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Most  of  the  instruments  are  used  on  an  'as  needed’  basis  or  irregularly.  Only  one  library 
reports  that  it  has  a biennial  user  satisfaction  surveys.  The  other  exception  is  evaluation  after 
library  instruction  on  electronic  materials  that  seems  under  represented  here.  Three  libraries 
mentioned  participation  in  the  ARL  LibQUAL+  project  as  an  effort  to  collect  information 
regarding  service  quality.  Focus  groups,  user  panel  feedback  and  user  forum  represent  important 
but  less  frequently  utilized  ways  of  collecting  data. 

In  addition  to  the  instruments  mentioned  in  Table  3,  there  were  a few  items  that  some 
libraries  are  measuring  about  electronic  resources.  One  library  is  collecting  dollar  amounts  saved 
(cost  avoidance)  as  part  of  the  consortium  report.  What  is  not  clear  is  whether  the  information  is 
the  cost  savings  at  that  institution  or  at  a consortium  level.  One  library  listed  the  number  of 
students  who  took  library's  online  tutorials  and  courses  as  a measure  in  the  other  category. 

Data  Use 


1)  Level  of  communication 

The  survey  requested  that  libraries  indicate  the  people  or  entities  with  whom  they  report  the 
collected  data  on  electronic  resources.  Table  4 summarizes  the  result  along  the  provided 
categories. 

Table  4.  Dissemination  of  Data 


Among  library 

Among  library  staff 

With  supporting 

Others 

managers 

institutions 

100% 

95% 

82% 

32% 

The  "other"  category  included  ARL  (2),  outside  accreditation  body  (1),  external  funding 
agency,  consortia  groups  (1),  magazine  (1)  and  public  (1). 

On  the  surface,  it  seems  information  on  electronic  resources  and  services  (to  the  extent  it  is 
available)  is  being  communicated  widely  within  the  library  and  within  the  parent  organization. 
Without  specific  information  about  what  kinds  of  information,  medium  (e.g.,  annual  reports, 
official  internal  report),  and  frequency,  it  is  difficult  to  ascertain  the  level  of  communication  that 
takes  place  for  measures  and  statistics  of  electronic  resources. 

2)  Use  of  Collected  Data  for  Decision-Making 

We  also  asked  the  libraries  to  elaborate  on  the  kinds  of  decision-making  based  on  collected 
measures  and  statistics.  To  help  summarize  the  responses,  we  developed  the  following  taxonomy 
in  Table  5. 

One  of  the  immediate  uses  of  measures  of  electronic  resources  is  in  the  area  of  decision 
making  related  to  external  resource  contracts.  Fourteen  libraries  specified  that  they  use  electronic 
resource  statistics  to  make  electronic  database  subscription  decisions.  Among  those  libraries,  8 
reported  that  they  use  tum-away  or  lock-out  (logins  exceeded  simultaneous  user  limit)  data  to 
change  the  simultaneous  user  (S/U)  licenses.  It  is  not  clear  whether  the  information  is  used  to 
both  increase  and  decrease  the  number  of  S/U  licenses.  Three  libraries  responded  that  usage 
statistics  of  electronic  resources  affect  the  subscription  decision  of  the  same  or  similar  counterpart 
print  materials. 
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In  the  reporting  and  communication  category,  seven  libraries  report  that  data  are  being  used 
for  budget  related  activities.  It  is  interesting  to  note  that  only  one  library  specifically  mentions 
the  use  of  such  data  in  their  annual  report.  The  library  wrote: 

The  university  administration  has  little  interest  in  frequent  statistical  reports,  but  we  make 
sure  that  the  overall  extent  of  our  e-resources  and  bottom-line  summaries  of  the  extent  of 
their  use  feature  prominently  in  our  most  significant  reports  [annual  report]  to  them. 

Table  5.  Taxonomy  for  Decision-Making  Instances  Affected  by  Information  on  the  Use  of 
Electronic  Collection. 


Main  Category 

Specific  Category 

Frequency 

External  Resource 

Journal  renewal/cancellation 

3 

Contract 

Database  renewal/cancellation 

14 

Changes  in  simultaneous/user  Limit 

8 

Reporting  and 

Budget  request,  justification,  and  presentation. 

7 

Communication 

Annual  report  and  other  similar  summary  report 

1 

Strategic  planning 

3 

Institutional  comparison 

1 

Service  Assessment  & 

Redesign  of  web  pages 

9 

Improvement 

Marketing  of  collections  and  services 

3 

Instruction  and  training 

4 

Changes  in  staffing 

4 

Assessment  of  existing  collections  and  services 

9 

Assessment  of  pilot  (trial)  collections  and 
services 

1 

Three  libraries  use  the  data  in  strategic  planning  process  and  related  documents.  There 
was  only  one  library  that  mentioned  specifically  institutional  comparison  as  one  of  the  uses  of 
the  data. 

In  the  service  assessment  and  improvement,  nine  libraries  said  they  use  the  data  to 
redesign  library  webpages.  We  suspect  the  high  number  of  responses  in  this  category  might 
have  been  influenced  by  the  example  given  in  the  question.  The  same  number  of  libraries 
responded  using  a more  generic  phrase  that  can  be  summarized  as  assessment  of  existing 
collections  and  services.  Only  one  library  said  that  it  uses  the  data  to  evaluate  pilot  services. 

There  were  hints  in  the  responses  that  some  of  the  decision  making  activities  are 
performed  only  on  occasion  (as  opposed  to  regularly  or  systematically),  in  an  ad  hoc  (as 
opposed  to  planned)  manner.  The  following  comment  from  one  library  will  illustrate  the 
point. 

At  this  point  in  time  we  have  neither  sufficiently  comprehensive  statistics  nor  the 
necessary  processes  in  place  to  use  these  statistics  to  systematically  evaluate  our 
collections  and  service. 

Nonetheless,  libraries  use  the  data  to  make  improvements  in  service  marketing  (3), 
instruction  & training  (4)  and  staffing. 
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Issues  Important  to  Statistics  and  Measurement 

The  survey  asked  the  respondents  to  list  the  three  most  important  issues  related  to  collection 
of  statistics  and  measures  to  describe  networked  resources  and  services.  Eighteen  libraries 
responded  to  the  question,  an  overwhelming  number  of  responses  were  related  with  problems 
associated  with  current  vendor  reports. 

Table  6 describes  the  summary  of  these  responses.  The  main  category  is  broken  into  two 
sections:  one  that  has  to  do  with  data  collection  and  the  other  with  data  processing  and  use.  The 
data  collection  category  is  further  divided  into  issues  related  with  vendor  reports  and  others. 

Table  6.  Summary  of  Issues  Related  to  Measures  of  Electronic  Resources  and  Services 


Main  Category 

Specific  Category  Frequency 

Data  Collection  In  regard  to 

vendor  statistics 

Lack  of  consistent  definitions  9 

Lack  of  comparable  measures  12 

Lack  of  standardized  reporting  method  1 0 

Lack  of  detailed,  granular  data  3 

Lack  of  useful  data  3 

Availability  of  data  2 

Timeliness  of  data  1 

Effort  to  collect  data  4 

Other 

Difficult  to  measure  web  access  1 

Difficult  to  aggregate  data  among  branch  1 

libraries 

Lack  of  qualitative  data  1 

Lack  of  information  about  users  3 

Data  Processing 
and  Use 

Lack  of  supporting  (technological  and  2 

human  resources)  to  facilitate  data 
processing 

Urgency  to  j ustify  expenditure  1 

Difficult  to  summarize  or  interpret  data  2 

Inability  to  relate  use  of  electronic  1 

materials  with  their  physical 
counterparts 

Comparison  with  peer  institutions  1 

Inability  to  link  data  to  decision  making  1 

Inability  to  link  data  to  quality  of  service  1 

Inability  to  link  data  to  outcomes  2 

The  data  clearly  show  that  first  of  all  libraries  want  consistent,  comparable  data  delivered  in  a 
standardized  method.  The  responses  read  like  a scripted  answer  whenever  you  ask  "what’s  wrong 
with  statistics  on  electronic  resources?"  They  suggest  that  vendors  are  to  blame.  A high  level  of 
frustration  with  the  contents  of  vendor  supplied  usage  reports  and  the  way  they  are  delivered  were 
common  responses.  One  respondent  summed  it  up  saying  "for  the  most  part,  vendors  provide 
what  they  want"  not  what  libraries  want. 

On  the  other  hand,  one  library  expressed  some  optimism  amid  frustration  about  the  vendor 
reports.  "Despite  these  frustrations,  vendors  are  making  progress  in  moving  towards  simpler  and 
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more  accessible  reports  and  we  are  pleased  that  for  our  most  important  resources  port  contention 
[turn  away]  is  reported  to  us." 

Four  libraries  expressed  concern  that  it  takes  too  much  time  and  effort  to  collect  and  compile 
various  vendor  statistics.  Some  pointed  out  the  issue  of  not  having  useful  data  (3)  or  data  with 
enough  detail  (3).  Two  libraries  said  some  vendors  provide  no  statistics  at  all.  One  library 
complained  that  vendor  reports  are  not  timely. 

Among  data  collection  issues  other  than  those  related  with  vendor  reports,  the  lack  of 
information  about  users  and  user  behaviors  tops  the  list  of  responses.  One  library  said  that 
aggregating  measures  across  branch  libraries  on  campus  is  a difficult  thing  to  do  given  the  fact 
that  some  libraries  operate  either  independently  or  under  different  reporting  environment. 

Another  library  expressed  concern  that  collecting  even  the  seemingly  simple  thing  such  as 
webpage  access  proved  to  be  a difficult  task  due  to  technological  and  organizational  complexities. 
Finally,  one  library  said  that  lack  of  qualitative  oriented  information  such  as  ease  of  access, 
quality  of  sources,  and  availability  is  an  issue. 


The  responses  suggest  relatively  low  responses  related  to  data  processing  and  use.  While  this 
may  have  to  do  with  the  way  the  question  was  phrased,  it  does  beg  the  question  "are  libraries 
doing  enough?"  At  least  two  libraries  commented  that  lack  of  organizational  support  in  terms  of 
technical  and  human  resources  is  a major  hindrance  to  the  measurement  effort.  Two  other 
libraries  said  they  are  having  difficulty  interpreting  and  summarizing  the  collected  data.  These 
kinds  of  responses  acknowledge  the  fact  that  even  when  libraries  receive  reasonably  satisfactory 
reports  from  database  vendors,  a lot  of  work  still  remains.  Four  libraries  pointed  out  their 
inability  to  link  data  to  other  important  issues  such  as  decision-making  (1),  quality  of  service,  and 
educational  outcome  (2). 


Willingness  to  Participate  in  the  Benchmarking  Process 

When  the  survey  asked  libraries  whether  libraries  are  interested  in  the  formal  benchmarking 
process,  we  provided  one  criterion  for  best  practice  that  there  has  to  be  a systematic  process  put  in 
place  for  a prolonged  period  of  time  (at  least  6 months). 

Table  7 shows  the  number  of  essentially  self-selected  best  practices  libraries  in  the  provided 
areas  of  expertise. 

Table  7.  Number  of  Self-chosen  Best  Practices  and  Their  Areas  of  Expertise 


Category 

Users  of  networked  services  and  resources 

Uses  of  networked  services  and  resources 

Quality  of  networked  services  and  resources 

Cost  of  networked  services  and  resources 

Staffing  and  training 

[Other]  Life  Cycle  Funding  Analysis 

[Other]  Training/programming  for  increased  use 


No.  of  self-selected  best  practice  libraries 
2 
6 
2 
3 
3 
1 
1 
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Summary  of  Survey  Findings 

Analysis  of  the  E-Metrics  survey  responses  reveals  a wide  range  of  data  collection  and  use 
activities  among  project  participants.  It  appears  that  measures  related  to  patron  accessible 
resources  and  costs  are  more  consistently  and  systematically  collected  than  measures  related  to 
electronic  resource  use  or  users  of  those  resources.  Due  to  often  inconsistent  and  non-comparable 
nature  of  vendor  supplied  statistics,  libraries  seem  to  have  a difficulty  to  track  overall  electronic 
database  usage  and  its  patterns. 

Collected  data  seem  to  be  shared  widely  among  library  staff  and  with  supporting  institutions. 
However,  the  types  of  media  information  is  communicated  and  the  nature  of  the  information 
sharing  are  now  shown  in  the  data.  Data  are  most  often  used  to  make  purchasing  decisions  for 
licensed  vendor  materials.  People  also  indicated  various  uses  of  collected  data  for  the  purpose  of 
internal  and  external  reporting  and  service  assessment  and  evaluation. 

Responding  to  what  are  the  most  important  issues  related  to  performance  measurement  of 
networked  resources  and  services,  the  majority  of  respondents  cite  the  lack  of  consistent  and 
comparable  statistics  from  database  vendors  as  the  most  serious  problem.  Relatively  few 
respondents  cite  problems  associated  with  the  library's  inability  to  process  and  utilize  collected 
data. 
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2.  Comparison  of  Vendor  Reports 

We  analyzed  the  sample  vendor  reports  for  12  different  vendors  to  investigate  what  they  offer 
and  how  they  deliver  reports.  As  we  explained  in  the  methodology  part,  the  vendors  were  chosen 
from  the  lists  submitted  by  the  members  of  the  ARL  Working  Group  on  Dadabase  Vendor 
Statistics.  Generally  speaking,  these  vendors  represent  a subset  of  expensive  and/or  most  widely 
used  databases  among  more  than  several  dozen  database  vendors  with  which  individual  libraries 
have  business  relationships.  The  1 2 vendors  represent  different  types  of  electronic  database 
services  available  on  the  market.  For  example,  Academic  Press/IDEAL  and 
Elsevier/ScienceDirect  are  providers  of  scholarly  journal  contents.  GaleNet  is  a provider  of 
indexing  and  reference  tools.  ISI/Web  of  Science  is  a citation  database. 

Comparing  vendor  statistics  reports  proved  to  be  a challenging  task.  First,  many  reports  were 
written  in  such  a way  that  it  was  difficult  to  decipher  what  exactly  the  reported  statistics  mean. 
Second,  the  reports  are  evolving  constantly  as  the  vendors  change  both  the  content  and  the  format 
in  attempts  to  improve  their  service.  We  advise  our  readers  to  refer  to  the  information  compiled 
by  the  Washington  State  Library  in  regard  to  current  and  planned  vendor  reporting  capabilities 
(attached  in  Appendix  D). 

The  analysis  consists  of  three  main  areas:  1)  Overall  comparison  of  reported  measures  and 
delivery,  2)  Availability  (or  lackjof  definitions,  and  3)  Vendor  compliance  to  the  ICOLC 
guidelines. 

The  purpose  of  the  following  comparisons  is  to  show  and  document  a wide  range  of  reporting 
practices  and  to  develop  strategies  in  planning  collaboration  with  major  vendors.  As  already 
pointed  out,  there  are  many  vendors  who  do  not  provide  any  statistics  at  all.  This  poses  a more 
serious  problem,  as  libraries  do  not  have  information  whatsoever  to  evaluate  the  effectiveness  of 
the  products.  Overall,  larger  vendors  are  more  likely  to  furnish  some  type  of  usage  statistics 
probably  because  they  have  technical  and  human  resources  to  generate  reports  while  smaller 
providers  lack  such  capabilities. 

Overall  Comparison  of  Vendor  Usage  Reports 

Table  8 summarizes  the  vendor  reports  in  terms  of  content  (reported  statistics,  statistics 
organization)  and  report  delivery  (frequency,  method  of  delivery,  and  format).  For  example, 
Academic  Press/IDEAL  provides  the  number  of  sessions  and  the  number  of  document  types 
(such  as  abstracts,  full-text  articles)  accessed  organized  by  journal  title.  The  report  is  compiled 
monthly  and  accessible  on  the  company's  website  in  Excel  spreadsheet  format.  In  some 
instances,  reports  are  available  in  two  different  methods.  The  ProQuest  usage  statistics  are 
available  both  in  print  and  on  the  web  (in  this  case,  libraries  can  request  the  detailed  report  sent  as 
an  email  attachment). 
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Table  8.  Comparison  of  database  vendor  reports 

Vendor 

Reported  Main  Statistics 

Organized 

By§ 

Report  Format 
& Delivery 

Frequency11 

Academic  Press 
IDEAL 

Sessions 

Document  types  accessed  or 
downloaded 

By  journal  title 

HTML  and  Excel,  Web 

Monthly 

and 

Quarterly 

Bell  & Howell 
ProQuest 

Documents  accessed  by  type 

By  journal  title 
and  by 
database 

Print 

Text,  Email  (Detailed) 
HTML,  Web  (Summary) 

Monthly 
or  range  of 
months 

Ebsco 

Searches 

Hits 

Documents  by  type 
Emails  sent 

By  database 
and  by  IP 
address 

HTML,  Web 
Text  or  HTML,  Email 

Monthly 
or  range  of 
months 

Elsevier 

ScienceDirect 

Searches 

Requests  of  webpages 
Journals  browsed 
Articles  accessed 
Help  files  accessed 

By  journal  title 
(Subscribed 
vs. 

Unsubscribed) 

MS  Word,  Email 

Monthly 

GaleNet 

Searches 
Entries  retrieved 
Help  files  used 

By  journal  title 

HTML,  Web 
Text,  Email 

Monthly 

HighWire 

Searches 

Document  accessed  by  type 
Top  10  articles  viewed 

By  journal  title 

HTML,  Web 

Monthly 

and 

Quarterly 

ISI 

Web  of  Science 

Queries 

Sessions 

Average  length 

Maximum  concurrent  users 

Turn-aways 

Not 

Applicable 

HTML,  Email 

Monthly 

JSTOR 

Searches 

Browsing  (by  sections  of 
record) 

Viewing  (articles,  pages) 
Printing  by  types  of  formats 

By  journal 
title, 

discipline,  IP 

HTML  and  text,  Web 

Monthly 

Lexis-Nexis 

Searches 

Documents  retrieved 

By  database 
By  day  and  by 
hour  of  the  day 

HTML  and  comma 
separated  (CSV)  file, 
Web 

Monthly 

OCLC 

FirstSearch 

Sessions 

Session  turn-aways 
Searches 

Documents  ordered 
Port  usage 

By  database 

HTML,  Web 

Monthly 

OVID 

Sessions 

Time 

Sets 

View,  print  and  save 

By  database 

HTML,  Web 

Can  be 
specified 
down  to  a 
minute 

Silver  Platter 

Total  connection  time 
Successful  logins 
Characters  delivered 
Items  browsed 

By  database 

Text,  Email 

Monthly 

§ Statistics  organization:  by  journal  title  (or  journal  name  by  some  vendors)  means  statistics  are  reported  at 
the  journal  or  newspaper  or  magazine  level,  by  database  means  statistics  are  reported  at  the  group  of 
journals  or  source  files. 

^Frequency  refers  to  the  report  interval.  Monthly  means  the  statistics  are, compiled  and  reported  monthly. 
Some  vendors  give  an  option  to  specify  the  range  of  request  period,  usually  in  months. 
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Findings  and  issues  identified  in  the  comparison  are  as  follows: 

• Many  vendors  offer  multiple  delivery  options.  While  the  HTML  format  is  easily  viewed 
in  a browser,  text  format  (ASCII  or  comma  separated  ) is  easier  to  manipulate  using  a 
spreadsheet  or  database  program.  In  some  cases,  libraries  were  not  informed  of  the  fact 
that  the  same  report  is  available  in  an  alternative  form.  The  case  in  point:  several 
libraries  sent  us  a print  copy  of  the  ProQuest  report  while  other  libraries  showed  us 
copies  of  statistics  obtained  from  the  vendor  website. 

• While  most  other  reports  provide  raw  statistics  only,  JSTOR  provides  institutional 
comparison  of  usage  based  on  the  size  of  institutions.  It  also  provides  graphs  showing 
usage  trends  over  time. 

• Although  not  included  in  the  comparison,  the  timeliness  of  usage  reports  was  identified 
as  one  of  the  key  issues.  While  majority  of  the  vendors  who  provide  monthly  report  do 
so  within  the  following  month,  some  vendors  lag  several  months  behind  and  in  some 
cases  reports  for  a block  of  months  are  not  available  due  to  the  technical  failures  of  the 
vendor  reporting  program.  It  is  strongly  recommended  that  the  availability  of  timely 
(need  to  be  specified)  report  included  in  the  licensing  contract. 

• When  database  vendors  provide  journal  level  statistics,  they  should  be  arranged  in  such  a 
way  (e.g.,  alphabetically)  to  facilitate  easy  manipulation.  If  a database  product  or  a 
journal  is  broken  into  several  different  parts,  as  in  the  case  of  Medline  having  several 
blocks  of  files  divided  by  the  database  coverage,  the  total  usage  for  that  database  or 
journal  needs  to  be  provided. 

• When  the  usage  statistics  are  available  on  the  web,  a user  ID  and  a password  are  required 
to  access  the  statistics.  JSTOR  is  an  exception.  It  does  not  require  a user  ID  and 
password.  Instead  anyone  whose  IP  address  belongs  to  the  eligible  IP  block  for  use 
authentication  can  request  and  retrieve  statistics.  Sometimes,  a separate  set  of  ID  and 
password  is  required  to  access  statistics  for  each  database  or  journal.  HighWire  is  an 
example  of  such  case.  Currently,  it  requires  libraries  to  use  the  same  user  ID  and 
password  to  access  statistics  as  well  as  administer  institutional  profiles. 

• Overall,  libraries  want  usage  statistics  that  are  easy  to  obtain  and  manipulate  if  necessary. 
In  other  words,  there  should  be  less  or  little  burden  on  the  library  to  process  the  statistics. 
Ideally,  vendors  need  to  use  standardized  measures  and  formats.  Another  issue  has  to  do 
with  different  needs  from  different  libraries.  It  is  strongly  recommended  that  a summary 
level  information  be  provided  to  all  libraries.  At  the  same  time,  detailed  and  granular 
statistics  should  be  also  available  to  libraries,  in  ASCII  (Text)  or  comma  separated  file 
(CSV)  format,  who  wish  to  obtain  additional  information  about  the  use  and  use  patterns. 


Provision  of  Definitions  of  Measures 

One  of  the  most  frequently  cited  complaints  about  the  current  vendor  reports  is  the  lack  of 
standardized  terms  for  key  usage  indicators  and  the  lack  of  definitions.  Table  9 shows  the 
availability  of  definitions,  descriptions  or  explanations  of  reported  measures.  It  clearly  shows 
that  many  vendors  do  not  provide  information  about  what  the  statistics  mean.  Even  when 
information  is  provided,  it  is  mere  repetitions  or  paraphrases  of  a measure  (see  examples  in 
HighWire,  JSTOR,  and  Silver  Platter) 
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Table  9.  Availability  of  Descriptions  of  Reported  Statistics 


Vendor 

Definitions  or  explanations 

Academic  Press/DDEAL 

Not  Available 

Bell  & Howell/ProQuest 

Not  Available 

Ebsco 

Not  Available 

Elsevier/ScienceDirect 

Not  Available 

GaleNet  (in  the  statistics 

• Searches  submitted:  This  category  tracks  the  number  of  searches 

web  page) 

that  you  have  run  in  this  database  for  the  month  selected.  Each 
time  you  submit  a search  it  generates  a single  "Search 
Submitted.  "If  you  modify  a search  and  resubmit  it  that  will 
generate  one  more  'search  run." 

• Entries  Retrieved:  This  category  counts  the  number  of  entries 
that  you  looked  at  in  this  database.  If  your  search  returned  15 
entries  in  the  search  results  list,  and  you  retrieved  two  by 
clicking  on  them,  you  would  generate  two  'entries  retrieved.'  If 
you  have  retrieved  an  entry  and  you  click  on  a hypertext  link 
within  that  entry  to  a related  entry,  clicking  on  that  hypertext 
link  will  generate  one  more  'entry  retrieved.' 

HighWire  (in  the  statistics 

• Searches:  The  number  of  searches  performed  on  the  search  page. 

web  page) 

• Abstracts  and  Full-text  HTML:  The  number  of  abstracts,  full- 
text  articles  in  HTML  format  viewed. 

• PDF's:  The  number  of  PDF  files  downloaded. 

ISI 

Not  Available 

JSTOR 

The  total  number  of  accesses  for  your  institution  is  divided  into 

(from  library  submitted 

several  categories: 

copy  of  JSTOR  web  page) 

• Browsing:  Accesses  which  may  indicate  browsing  behavior. 

• Viewing:  Accesses  which  reflect  that  article  pages  were  viewed 
online. 

• Printing:  Number  of  articles  printed  or  downloaded  from  the 
printing  page  in  a special  printing  format. 

• Searches:  Number  of  searches  performed  in  all  journals,  and  the 
number  of  search  results  lists  viewed. 

• Total:  Total  number  of  accesses  (browsing,  viewing,  printing 
and  searches  combined). 

Lexis-Nexis 

Not  Available 

OCLC  - FirstSearch 

Not  Available 

Ovid 

Not  Available 

Silver  Platter 

Explanation  of  Statistics 

(in  the  email) 

• Time:  total  “connection”  time  to  the  database. 

• Num:  total  number  of  successful  logins  to  the  database. 

• Chars:  total  number  of  characters  delivered. 

• Without  abstract:  total  number  of  records  browsed  without  an 
abstract. 

• Total  records:  total  number  of  records  browsed  (including 
records  with  an  abstract). 
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Level  of  Vendor  Compliance  to  the  ICOLC  Guidelines 

The  ICOLC  guidelines  are  widely  recognized  as  the  acceptable  practice  by  both  the  library 
and  vendor  communities.  The  guidelines,  drafted  in  November  1998  by  the  International 
Coalition  of  Library  Consortia  (ICOLC),  have  the  following  five  use  elements  that  a vendor 
report  needs  to  provide: 

1.  Number  of  queries  (Searches)  categorized  as  appropriate  for  the  vendor’s  information.  A 
search  is  intended  to  represent  a unique  intellectual  inquiry.  Typically  a search  is  recorded  each 
time  a search  form  is  sent/ submitted  to  the  server. 

2.  Number  of  Menu  Selections  categorized  as  appropriate  to  the  vendor’s  system.  If  display  of 
data  is  accomplished  by  browsing  (use  of  menus),  this  measure  must  be  provided  (e.g.  an 
electronic  journal  site  provides  alphabetic  and  subject-based  menu  options  in  addition  to  a search 
form.  The  number  of  searches  and  the  number  of  alphabetic  and  subject  menu  selections  should 
be  tracked). 

3.  Number  of  sessions  (Logins),  if  relevant,  must  be  provided  as  a measure  of  simultaneous  use.  It 
is  not  a substitute  for  either  query  or  menu  selection  counts. 

4.  Number  of  tum-awavs,  if  relevant,  as  a contract  limit  (e.g.,  requests  exceed  simultaneous  user 
limit). 

5.  Number  of  items  examined  (i.e.,  viewed,  marked  or  selected,  downloaded,  emailed,  printed)  to 
the  extent  these  can  be  recorded  and  controlled  by  the  server  rather  than  the  browser: 

Table  10  provides  an  estimated  compliance  level  for  each  vendor  included  in  the  comparison. 

Due  to  the  lack  of  definitions  and  the  exact  nature  of  licensing  and  database  characteristics,  it  is 
impossible  at  this  point  to  complete  the  table.  Nonetheless  it  shows  that  this  group  of  vendors 
appear  to  provide  a subset  of  core  measures  closely  aligned  with  the  ICOLC  guidelines.  There  are 
several  cases  where  vendors  (e.g.,  Lexis-Nexis  and  GaleNet)  expressed  future  plan  to  comply 
with  the  guidelines.  The  fact  that  a vendor  has  more  checkmarks  (S)  than  other  vendors  should 
not  be  interpreted  as  an  evidence  that  the  vendor  provides  a superior  report.  There  are  other 
factors  (report  delivery  and  the  level  of  specificity)  that  affect  the  quality  of  the  reports. 


Table  10.  Estimated  Vendor  Compliance  to  the  ICOLC  guidelines 


\ ICOLC  items 

Sessions 

Queries 

Items 

Tum-aways 

Menu 

Vendor  \ 

(Logins) 

(Searches) 

examined 

selections 

Academic  Press/IDEAL 

S 

n/a 

Bell  & Howell/ProQuest 

S 

n/a 

Ebsco 

S 

n/a 

Elsevier/ScienceDirect 

V 

n/a 

✓ 

GaleNet 

V 

V 

n/a 

HighWire 

V 

S 

n/a 

ISI 

S 

✓ 

JSTOR 

S 

V 

n/a 

Lexis-Nexis 

V 

n/a 

OCLC/  FirstSearch 

V 

V 

Ovid 

S 

V 

n/a 

Silver  Platter 

V 

S : the  measure  is  available. 

n/a  : not  applicable.  Some  databases  do  not  have  simultaneous  user  limit.  Hence  no  tum-away. 
Empty  cells  mean  either  not  available  or  there  is  not  enough  information. 
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One  item  that  may  need  consideration  for  future  consideration  is  connection  time.  Both  Ovid 
and  Silver  Platter  report  this  measure. 


Summary  of  Vendor  Reports  Comparison 

Analysis  of  usage  statistics  from  12  major  database  vendors  reveal  that  there  is  a wide  range 
of  different  practices  and  that  progress  should  be  made  in  several  areas  including  standardization 
of  core  statistics,  report  delivery  method,  and  assuring  the  provision  of  definitions  of  reported 
statistics.  There  are  some  signs  in  the  way  vendors  report  data  that  indicate  some  vendors  are 
moving  toward  the  right  direction.  One  of  the  signs  is  the  apparent  willingness  of  many  vendors 
to  comply  with  the  ICOLC  guidelines  (see  Appendix  D for  vendor  reporting  capabilities). 
Coordination  of  library  efforts  seems  necessary  to  bring  about  positive  change  in  reporting 
practices  and  develop  consistent  and  comparable  statistics. 
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3.  SITE  VISIT  REPORT 

The  following  summary  of  the  site  visits  is  organized  into  four  parts: 

(1)  Overview  of  library  facts,  which  provides  a brief  introduction  to  the  institution  and  a 
select  set  of  library  statistics; 

(2)  Overall  approach/philosophy  toward  data  collection  and  use,  which  highlights  the 
atmosphere  and  existing  strategies; 

(3)  Data  collection  activities,  which  deals  with  who  is  involved  in  the  process  and 
describes  their  roles;  and 

(4)  Data  use  and  main  users,  which  examines  the  main  consumers  of  the  collected 
information  and  the  documents  or  channels  through  which  data  are  disseminated. 


Virginia  Tech  (VPI  & SU)  Library 


1)  Overview 

Virginia  Polytechnic  Institute  and  State  University,  popularly  known  as  Virginia  Tech,  is  a 
comprehensive  university  located  in  Blacksburg,  VA  with  a national  reputation  for  its  premiere 
engineering  and  technology  programs.  It  is  the  largest  higher  education  institution  in  the  state  of 
Virginia  with  25,  000  students.  Key  statistics  about  the  library  are  summarized  in  Table  1 1. 

Table  II.  VT  Library  Statistical  Summary 


1997 

1998 

1999 

Total  Expenditure  (in  mil.) 

11.3 

11.1 

11.3 

Materials  Expenditure  (in  mil.) 

5.7 

5.4 

5.5 

E-Resources  Expenditure  (in  mil) 

0.5* 

0.5* 

0.5* 

Professional  Staff 

37 

37 

36 

Support  Staff 

97 

98 

98 

Total  Fulltime  Students 

23,624 

24,150 

24,550 

Total  Fulltime  Graduate  Students 

3,673 

3,702 

3,686 

Fulltime  teaching  Faculty 

1,410 

1,414 

1,203 

* : The  figure  does  not  include  the  VIVA  consortium  funding  from  the  state. 


2)  Overall  Approach/Philosophy  Toward  Data  Collection  and  Use 

Staff  indicated  that  the  main  reason  why  the  VT  Library  is  engaged  in  collecting  data  about 
its  collections  and  services  is  to  answer  the  question  "what  is  going  on  here?"  The  question, 
although  very  open-ended,  demonstrates  that  electronic  materials  and  services  are  relatively  new 
and  therefore  they  don't  have  a full  understanding  of  how  these  new  services  are  being  received 
by  users  and  whether  the  resources  are  returning  values  to  the  users  and  the  library. 

The  other  main  reason  why  the  library  collects  data  is  to  use  them  as  a vehicle  to  demonstrate 
library's  accomplishments  to  the  administration  and  the  user  community.  Data  are  gathered  with 
a reasonably  clear  understanding  of  how  they  will  be  utilized  in  various  reports  and  other 
justification  materials,  i.e.,  budget  requests. 

Having  the  right  data  in  this  area  has  become  an  important  issue  because,  like  many  other 
ARL  libraries,  VT  Libraries  is  experiencing  a steady  decline  in  services  such  as  reference, 
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external  circulation,  and  in-house  use  of  materials.  Knowing  how  users  are  meeting  their 
information  needs  became  an  important  area  of  investigation  for  the  library.  At  the  same  time, 
library  staff  want  to  know  the  dynamics  between  the  use  of  traditional  print  materials  and 
materials  in  digital  format  to  be  able  to  answer  questions  such  as  "is  there  a trade  off  between  use 
of  these  materials?” 

The  general  approach  at  the  VT  Library  in  terms  of  collecting  data  on  electronic  resource  use 
is  to  get  just  enough  data  that  can  support  "best  guess"  decision  making  for  service  delivery.  The 
data  also  helps  the  library  document  the  high  use  of  electronic  resources  to  justify  the  direction 
that  it  has  taken  and  demonstrate  that  with  more  funds  it  could  provide  more  services.  However, 
the  amount  of  institutional  resources  to  collect  and  process  more  detailed  data  is  quite  limited. 

First  and  foremost,  they  need  data  collection  techniques  that  are  easy  to  implement  and  use, 
preferably  in  formats  like  a spreadsheet  or  database.  They  also  would  prefer  data  that  can  be 
generated  in-house  rather  than  relying  on  vendors.  Usage  statistics  from  database  vendors  tend  to 
be  slow  in  coming  and  are  at  best  confusing  with  problems  such  as  inconsistency  of  measures  and 
unclear  definitions. 

The  number  of  click-throughs  to  individual  databases  and  full-text  journals  from  the  library’s 
electronic  database  webpage  is  a good  example  of  such  data  that  can  be  obtained  with  reasonable 
efforts.  VT  Library  did  and  still  can  collect  information  about  where  the  users  come  from, 
through  the  modem  pool,  from  somewhere  else  on  campus,  or  from  library  machines.  However, 
with  the  locations  constantly  changing,  they  are  no  longer  analyzed. 

3)  Data  Collection  Activities 

VT  Library's  data  collection  can  be  described  as  a distributed  model.  Three  key  staff 
members  are  currently  involved  in  the  process: 

• Director  of  Collection  Management  and  College  Outreach  for  use  data  from  database 
vendor  products  accessed  though  the  VIVA  (The  Virtual  Library  of  Virginia)  Consortium 
and  databases  mounted  locally; 

• Director  of  Information  Systems  for  generating  measures  related  with  library  webpage 
access;  and 

• Director  of  the  Digital  Library  and  Archives  (DLA)  for  use  statistics  related  with 
electronic  theses  and  dissertations  database,  and  electronic  journals  hosted  by  VT 
Library. 

VIVA,  started  in  1995,  is  a state-funded  consortium  aimed  at  cost  effective  purchase  and 
consolidated  access  to  a wide  array  of  selected  electronic  resources.  Thirty-nine  state-assisted 
universities  and  colleges  as  well  as  32  private  institutions  in  the  state  of  Virginia  now  participate 
in  VIVA.  The  state  funded  close  to  4 million  dollars  for  VIVA  related  expenditures  during  the 
1998-2000  period.  The  main  rationale  for  this  type  of  arrangement  includes  cost  savings  and 
making  resources  available  to  a wider  pool  of  institutions  which  could  not  have  afforded  access  to 
these  materials.  (VT  Library’s  electronic  materials  expenditure  is  significantly  underestimated 
since  it  does  not  include  the  VIVA  funded  portion.) 

As  far  as  facilitating  data  gathering  for  member  libraries,  VIVA  provides  a central  webpage 
containing  use  statistics  from  the  vendors.  With  a single  login  to  the  page,  member  libraries  can 
access  statistics,  both  current  and  past,  summarized  usually  by  month.  Note  that  the  data 
elements  contained  in  the  page  are  no  different  from  the  data  supplied  by  the  vendor  to  individual 
libraries  that  do  not  belong  to  the  consortium. 
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One  clear  advantage  of  belonging  to  a consortium  like  VIVA  is  that  libraries  can  save  time  in 
processing  the  raw  report  files  provided  by  database  vendors.  But  an  even  more  attractive  feature 
is  that  libraries  can  compare  their  own  numbers  with  statistics  for  other  peer  member  institutions. 
That  enables  VT  Library  to  ask  questions  such  as  "Why  is  VT  Library’s  use  of  GaleNet  so  low 
compared  to  U of  Virginia's?"  Therefore,  the  information  provides  ways  of  putting  data  into 
some  context  for  rudimentary  benchmarking  and  validating  common  senses. 

But  not  all  databases'  usage  statistics  are  summarized  by  institution  in  the  VIVA's  statistics 
page.  In  some  cases,  member  libraries  have  to  access  the  database  vendor's  statistics  webpage 
directly  to  retrieve  its  own  use  data  using  a separate  user  id  and  password  assigned  to  them  by  the 
vendor.  In  that  case,  they  can  not  have  comparison  data  unless  the  database  vendor,  for  example 
JSTOR,  provides  such  a feature. 

Below  is  the  list  of  vendors  whose  statistics  are  summarized  on  the  VIVA's  statistics  page. 

• ABC-CLIO 

• Cambridge  Scientific  Abstracts 

• Chadwyck-Healey 

• Congressional  Universe 

• Dow  Jones  Interactve 

• FirstSearch 

• GaleNet 

• InfoTrac 

• HarpWeek 

• Academic  Press/IDEAL 

• MathSciNet 

• Ovid 

• Oxford  English  Dictionary 

• Periodicals  Contents  Index 

• Project  Muse 

• Statistical  Universe 

• Stat-USA. 

An  institution  like  VT  has  many  electronic  databases  that  it  subscribes  to  in  addition  to 
databases  subscribed  through  VIVA.  Collecting  usage  data  from  these  databases  falls  to  the 
responsibility  of  the  Director  of  Collection  Management  and  College  Outreach.  He  uses  an  Excel 
spreadsheet  to  organize  information  such  as  vendor  statistics  webpage,  user  id,  password,  contact 
person  that  are  necessary  to  access  and  process  vendor  statistics. 

Collecting  usage  data  from  various  sources  such  as  VIVA,  vendor  websites,  saved  email 
messages  from  vendors  containing  reports  usually  takes  one  full  day  per  month.  He  noted  the 
difficulty  in  delegating  the  responsibility  to  other  people.  This  has  to  do  with  the  fact  that  vendor 
statistics  change  constantly.  There  are  not  yet  normalized.  It  requires  significant  background 
information  to  know  what's  going  on  with  the  vendors  and  what  various  statistics  at  different 
times  mean.  He  also  noted  that  it  is  difficult  to  delegate  the  data  collecting  activity  because  of  the 
necessity  for  someone  more  close  to  the  issue  on  hand  to  review  the  data  and  spot  any 
irregularities. 

All  the  collected  data  are  entered  into  an  Excel  spreadsheet,  mostly  on  a monthly  basis.  In 
the  spreadsheet,  each  database  file's  usage  is  represented  by  one  (two  in  a few  cases)  measure(s) 
most  indicative  for  that  particular  database.  While  for  most  database  products,  it  is  either  number 
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of  searches  (or  queries)  or  number  of  sessions  (or  logins),  other  units  such  as  item  views,  hits  and 
connection  hours  are  being  summarized  as  well.  The  compiled  data,  though  with  some  gaps, 
provide  the  usage  trend  over  time. 

VT  Library  has  a unit  called  Digital  Library  and  Archives  (DLA)  that  has  the  following  three 
major  electronic  information  sources: 

• VT  theses  and  dissertations  (2,356  items  as  of  6/30/2000); 

• Electronic  full-text  journals  mounted  on  and  accessed  through  VT  (21  journals  as  of 
6/30/2000);  and 

• VT  Imagebase. 

Usage  statistics  such  as  page  requests,  for  these  groups  of  collections  are  being  captured  by  a web 
access  log  software  called  analog  v4.1 1 on  a monthly  basis.  For  full-text  journals,  page  request 
information  by  journal  is  available.  A summary  report  (98/99  and  99/00)  is  available  on  the 
homepage  of  DLA  (http://scholar.lib.vt.edu).  The  log  analysis  part  of  the  report  is  essentially 
automated.  Once  the  data  are  available,  they  are  then  keyed  into  an  Excel  file  to  generate  various 
graphs  showing  monthly  and  yearly  trends. 

4)  Data  Use  and  Main  Users 

In  regard  to  licensed  materials  included  in  the  VIVA  package,  there  is  a selection  committee 
that  carefully  monitors  all  renewals.  Materials  specific  to  VT  are  also  monitored  monthly  by  the 
Director  of  Collection  Development  and  the  University  Librarian. 

Figure  1.  VT  Library  Electronic  Journals  Webpage. 


Data  on  database  usage  is  scrutinized  for  contract  renewal/cancellation.  Cancellation  of  a 
database  occurs  only  after  a well-publicized  and  intensely  studied  process. 
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The  data  are  being  used  to  improve  service  advertising  or  delivery.  A specific  example  is  that 
the  VT  Library  keep  monitoring  data  that  captures  what  users  do  in  the  library’s  electronic 
journals  page  as  shown  in  Figure  1. 

One  of  the  things  library  staff  discovered  from  the  data  is  that  many  users  confuse  the  search 
box  for  the  online  journals  as  an  Internet  search  engine  or  a search  box  for  the  library  online 
catalog  (OPAC).  There  have  been  several  attempts  to  fix  the  problem.  One  of  the  solutions  was 
to  place  a separate  link  to  the  OPAC  in  the  same  page  so  that  people  know  they  have  to  click  on 
that  link  to  go  to  a page  where  they  can  search  the  OPAC.  The  finding  also  helped  the  staff 
involved  in  user  education  to  emphasize  the  differences  among  various  search  aids  or  tools. 

A primary  medium  used  for  communication  with  the  university  administration  is  the  annual 
report.  In  the  report,  the  overall  extent  of  electronic  resources  and  overall  summary  of  the  extent 
of  their  use  are  featured  prominently.  The  choice  of  data  to  be  included  in  the  report  and  the 
narratives  that  go  along  with  the  data  are  crafted  by  the  University  Librarian  with  consultation 
from  directors  involved  in  the  particular  data  elements.  The  following  table  summarizes  the 
kinds  of  information  included  in  the  annual  report  during  the  last  three  years. 

Table  12.  Data  Related  with  Electronic  Resources  Appear  in  VT  Library  Annual  Reports. 


1997/98 

1998/99 

1999/00 

Top  ten  databases 
No.  of  electronic  (reference) 
questions 

Top  ten  databases  (with  # of 
searches) 

Articles  delivered  through 
online  databases 
Accesses  to  Digital  Library  and 
Archives  Resources 

# of  online  databases 

# electronic  journals 

Top  ten  databases  (with  # of 
searches) 

Full  text  articles  delivered 
from  major  vendors 

# of  electronic  reference 

service 

Library  web  hits 

# of  full-text  electronic 

journals 

Table  12  shows  that  over  the  years  the  number  of  reported  measures  is  increasing.  While  in 
97/98  only  the  list  of  top  10  databases  are  mentioned,  in  98/99  and  99/00  the  list  also  provides  the 
number  of  searches  conducted  by  users.  The  number  of  documents  delivered  is  also  featured  in 
the  report  as  an  indication  of  more  refined  measure  of  user  benefits. 

Sometimes  the  information  is  presented  with  an  annotation  that  gives  a hint  of  how  VT 
Library  is  fulfilling  user  demands.  For  example,  referring  to  the  inclusion  of  the  Web  of  Science 
from  ISI  (Institute  of  Scientific  Information)  in  the  top  ten  database  list,  the  98/99  report  notes, 
that  the  library: 

Procured  a five-year  backset  of  the  Web  of  Science,  allowing  Virginia  Tech  faculty  and 
researchers  to  follow  the  flow  of  ideas  and  influence  among  scholarly  papers  via  citation- 
based  relationship  and  noted  17,000  uses  of  this  new  service. 

5)  Summary 

VT  Library's  measurement  efforts  in  describing  networked  resources  can  be  summarized  as 
’’opportunistic”  or  on  an  as  needed  basis.  Data  collection  activities  appear  to  be  closely  related 
with  decision-making.  The  library  director  is  actively  engaged  in  the  evaluation  and  reporting 
process.  Also  it  appears  that  collected  data  are  being  used  in  library  reporting  documents.  The 


38 


67 


ARL  E-Metrics  Project  Phase  One  Report 


fact  that  the  library  has  a highly  visible  local  digital  collection,  electronic  theses  and  dissertations, 
gives  an  impression  within  and  outside  the  university  community  that  it  is  an  effective  player  in 
the  electronic  environment.  The  consortium's  role  as  a facilitator  of  data  collection  was  visible 
and  enhances  the  electronic  presence. 


University  of  Pennsylvania  Library 


1)  Overview 

University  of  Pennsylvania  is  an  urban  IVY  league  school  located  in  West  Philadelphia  and  it 
is  known  for  strong  programs  in  such  fields  as  business,  medicine,  and  communication.  Key 
statistics  about  the  library  are  summarized  in  Table  13. 

Table  13.  U.  Penn's  Library  Facts 


1997 

1998 

1999 

Total  Expenditure  (in  mil.) 

26.3 

28.8 

29.7 

Materials  Expenditure  (in  mil.) 

8.5 

9.2 

9.7 

% of  Electronic  Resource 

7.6 

13.4 

13.2 

Professional  Staff 

110 

112 

117 

Support  Staff 

178 

173 

178 

Total  Students 

17,595 

17,595 

17,743 

Graduate  Students 

8,111 

8,111 

8,177 

Faculty 

2,060 

2,110 

2,165 

2)  Overall  Approach/Philosophy  Toward  Data  Collection  and  Use 

One  thing  that  may  distinguish  U.  Penn  library  from  many  other  ARL  libraries  is  that  the  bulk 
of  library  costs  (85%  during  98/99  year)  are  charged  back  to  12  schools  within  the  university 
system.  Started  in  the  1970s,  the  allocated  costs  system  places  the  library  under  constant  pressure 
to  justify  its  expenditure  and  its  cost  effectiveness  to  the  university  community.  Currently  the 
costs  are  allocated  to  schools  based  on  the  number  of  faculty  in  each  school  and  the  number  of 
course  units  offered  by  each  school. 

Perhaps  the  most  revealing  fact  attesting  to  the  library's  reporting  obligation  to  the  university 
administration  is  the  following  memo  from  the  Council  of  Deans  to  the  Director  of  Libraries. 
Here  the  deans  request  quantitative  information  about  the  use  of  library  resources  and  patterns  of 
information  need  among  students  and  faculty.1 

1.  Who  uses  the  library,  i.e. --breakdown  by  school;  status  (student/standing 
faculty/research  faculty/adjuncts/staff/full  vs.  part-time). 

2.  What  types  of  uses--books,  journals,  reference,  data,  study  space,  etc. 

3.  Intensity  of  use-time  and  volume  measures  for  both  onsite  and  remote  uses. 

4.  Relationships  among  the  above-type  of  user  by  type  of  use  by  intensity,  etc. 

5.  Patterns  in  the  above  over  time. 

6.  Comparisons  to  peer  institutions  re  the  above. 


1 The  study  team  asked  the  project  participants  whether  they  have  received  similar  questions.  Out  of  16 
libraries  responded,  only  3 said  they  received  a smaller  set  of  questions,  primarily  about  comparisons  to 
peer  institutions. 
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7.  Pattern  s/trends  in  intellectual  areas  of  interest  to  users. 

8.  Pattems/trends  in  information  resource  use  beyond  the  library— where  do  Penn  people 
go  for  what  types  of  information  and  how. 

9.  How  other  universities  develop  library  budgets  and  apportion  costs  among  components. 
Some  data  produced  by  the  library  in  response  to  the  query  are  available  in  Appendix  E. 

Another  gauge  of  the  importance  of  assessment  for  Penn  is  found  in  the  library’s  current 
strategic  plan  which  includes  this  goal  regarding  measurement:  ’’Develop  new  ways  of 
conceptualizing  and  measuring  library  effectiveness  based  on  outputs  and  measures  of  excellence 
to  promote  within  the  Library  growing  accountability  and  responsiveness  to  changing  information 
environments  and  to  track  progress  toward  our  goals."  At  both  the  strategic  and  operational 
levels,  Penn’s  Vice  Provost  and  Director  of  Libraries  encourages  a culture  that  relies  on  empirical 
data  for  decision-making  and  to  demonstrate  accountability.  While  we  did  not  find  a specific 
managerial  structure  dedicated  to  assessment,  we  did  observe  a high  awareness  of  its  importance 
among  the  range  of  staff— from  the  library  director  to  programmers— who  participated  in  the  site 
visit  discussions. 

U.  Penn’s  gives  a good  deal  of  thought  and  attention  to  the  presentation  of  data  and 
information  about  its  libraries.  Many  official  library  publications,  print  or  on-line,  are  of  high 
quality.  Two  examples  are  worth  mentioning  in  this  regard.  One  is  called  the  Penn  Library 
Resource. Guide,  an  annual  compendium  of  information  and  human  resources  available  to  the 
users  (html  version  is  located  at 

http://www. libra ry.upenn.edu/services/publications/library/guide01.pdf).The  other  one  is  a short 
brochure  called  Penn  Library  Facts  (annual  publication)  that  contains  summary  of  many  statistics 
regarding  library  collection,  services  and  finance 
(http://www.library.upenn.edu/services/publications/librarv/factsOO.pdf). 


3)  Data  Collection  Activities 

At  Penn,  the  library  takes  a team  approach  to  data  collection  and  analysis.  The  following  staff 
are  key: 

• The  Executive  Assistant  to  the  Vice  Provost  and  Director  of  Libraries  is  responsible 
for  the  discovery,  integration,  processing,  and  analysis  of  data  on  library  use, 
operations  and  services.  He  works  closely  with  senior  administration  and  line  staff  to 
identify  data  sources  and  statistical  needs.  He  does  not  own  the  majority  of  data 
collection  processes,  but  functions  instead  as  a liaison  between  important  consumers 
of  data  within  administration  and  the  functional  units  responsible  for  library  services 
and  programs.  He  is  also  responsible  for  the  design  and  publication  of  quantitative 
information  for  the  system. 

• Library  Systems  Office  manages  data  logs  generated  by  the  OPAC  and  other  web 
servers,  and  provides  software  programming  to  extract  data  for  analysis 

• The  Library  Web  Manager  is  responsible  for  many  of  the  infrastructural  features  of 
the  Library  web  that  provide  information  about  use.  A good  example  is  the  creation 
of  a click-through  scheme  that  is  used  to  measure  database  log-ons.  (See  more  on  this 
below.)  He  also  has  responsibility  for  configuring  and  managing  log  analyzer 
software. 
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• The  Director  of  Lippincott  Library  of  the  Wharton  School  provides  statistical 
expertise  and  analysis  performed  in  SAS  and  other  statistical  software  applications. 
He  is  a highly  experienced  internal  consultant  to  the  overall  data  analysis  effort. 

• The  Electronic  Acquisitions  Librarian,  functions  as  a main  contact  with  database 
vendors  and  supplies  usage  reports  to  the  executive  assistant. 

• Library  Assessment  Intern:  A graduate  internship  that  helps  with  the  organization 
and  management  of  log  files,  automated  routines  for  normalizing  streams  of  log  data, 
and  analysis  of  log  output.  A graduate  student  in  the  department  of  Computer  and 
Information  Science  in  Penn’s  School  of  Engineering  currently  holds  this  position. 

One  of  the  most  prominent  features  of  electronic  resource  use  at  U Penn  is  what  it  calls  the 
attempted  log-ons  to  various  licensed  materials.  The  information  is  captured  by  a program 
counting  the  click-throughs  on  the  library  webpage  developed  by  the  library  Web  Manager. 

The  attempted  log-ons  are  broken  down  into  IP  (Internet  Protocol)  address  domains  that 
belong  to  schools  as  well  as  the  library  and  modem  pool  domains  (see  tables  in  Appendix  E). 

The  result  is  an  approximate  picture  of  where  the  requests  to  licensed  databases  are  made.  Of 
course,  these  are  "attempted”  log-ons. 

The  measure  “attempted  log-on”  has  a few  advantages:  it  is  available  locally  and  thus  gives 
the  library  some  independence  from  database  vendors.  However,  the  information  it  provides  is 
spare.  While  it  offers  some  degree  of  demographic  information  and  allows  chronological  analysis 
of  use,  it  still  only  represents  the  number  of  requests  for  a database,  in  essence  the  number  of 
times  a link  is  clicked.  It  will  not  tell  if  a request  resulted  in  a session  with  the  database,  or  how 
the  database  was  used  during  a session. 

Like  other  libraries,  U.  Penn  Library  keeps  track  of  data  provided  by  database  vendors.  Since 
the  creation  of  an  Electronic  Acquisitions  Librarian  position,  Penn  has  improved  the  management 
of  licensing  agreements  and  streamlined  data  collection  from  those  vendors  who  provide  use 
reports.  While  this  organizational  change  has  helped  with  the  harvesting  of  use  information,  the 
Electronic  Acquisitions  Librarian  continues  to  be  dismayed  by  the  absence  of  use  reporting  in 
contract  agreements  and  the  failure  of  vendors  to  supply  ICOLC  recommended  data  elements  or 
even  more  modest  use  indicators. 

U.  Penn  Library  developed  a matrix  that  clearly  shows  which  vendors  are  providing  and 
which  are  not,  the  types  of  measures  reported  by  the  vendors  (searches,  connection  time,  log-ons, 
documents  viewed,  hits/pages,  and  bytes  sent).  (See  Appendix  E for  the  matrix)  The  Electronic 
Acquisitions  Librarian  is  responsible  for  collecting  usage  statistics  from  vendors  and  passing  the 
information  to  the  executive  assistant.  In  the  98/99  period,  U Penn  Library  reports  the  number  of 
searches  (either  searches  or  log-ons  from  the  vendor  statistics)  for  70  of  122  databases. 

U Penn  Library  also  maintains  information  about  the  number  of  "accesses"  to  the  library  web 
pages.  It  indicates  that  the  number  of  accesses  reported  excludes  button,  icons,  banners  and  other 
items  incidental  files. 

U Penn  Library  has  resources  other  ARL  libraries  may  not  have.  For  example,  it  has  a human 
resource  pool  that  contributes  to  data  collection  and  analysis.  In  addition  to  systems 
programmers,  it  gets  help  from  the  director  of  Lippincott  library,  the  business  library  for  the 
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Wharton  School.  As  mentioned  above,  this  person  provides  knowledge  of  SAS  , a sophisticated 
statistical  analysis  program.  He  also  acts  as  a catalyst  of  ideas  that  can  give  new  angles  to  the 
collected  data  and  presentation.  For  example,  he  brought  to  the  meeting  a three-dimensional 
factor  loading  graph,  which  shows  the  relative  distances  among  Penn  schools,  represented  by  the 
types  electronic  databases  used  by  users  of  respective  schools.  This  particular  analysis  was  based 
on  data  harvested  from  attempted-logons.  Penn’s  School  of  Applied  Engineering  and  Science  is 
also  proving  to  be  a good  source  for  highly  skilled  students  who  can  supplement  the  efforts  of  full 
time  staff  in  the  development  of  measurement  strategies. 

Penn  staff  cited  the  following  as  having  significant  impact  on  their  efforts  to  measure  use  of 
e-resources: 


• the  lack  of  common  metrics  across  database  products  and  systems, 

• the  absence  of  use  measures  for  many  products, 

• the  great  time  required  to  chase  down  and  process  third  party  data  reports,  and 

• the  lack  of  a standard,  uniform  means  of  data  transmission.  Here  the  Penn  staff  was 
particularly  insistent  about  vendors  providing  a steady  stream  of  raw  or  semi- 
processed  use  data  from  logs  which  libraries  could  access  and  analyze  according  to 
their  local  needs  and  desires. 

Overall,  U Penn  Library's  data  analysis  capability  is  considered  exceptional  compared  to 
other  libraries.  Also  there  is  an  attitude  of  "are  there  any  other  ways  to  collect  and  look  at  the 
data?"  that  has  been  cultivated  in  the  library.  In  addition  to  asking  what  are  the  measures  that  can 
provide  quick  answers  about  electronic  materials  usage,  people  seem  to  be  quite  interested  in 
spending  extra  effort  to  produce  quality  data. 

The  case  in  point  is  analysis  of  data  provided  in  the  proxy  server  log.  Proxy  servers  allow 
users  who  connect  to  the  licensed  materials  through  various  ISPs  (internet  service  providers)  a 
legitimate  way  to  use  them.  With  the  introduction  of  a proxy  server,  it  is  possible  to  capture 
every  movement  a user  makes  in  a browser  (including  keystrokes  and  clicks)  after  the  point  when 
he  or  she  is  validated.  While  it  is  an  enormously  useful  data  source  not  only  about  use  statistics 
but  also  about  user  behaviors  which  otherwise  can  not  be  captured,  it  presents  a considerable 
burden  to  a library  to  sift  through  the  log  files,  which  can  be  quite  large.  Currently  U Penn 
Library  is  in  the  process  of  developing  software  programming  and  an  analysis  infrastructure  to 
tap  into  that  data  source. 

4)  Data  Use  and  Main  Users 

Usage  information  of  electronic  materials  is  being  used  for  budget  planning  and  justification. 
Information  is  provided  to  the  university  administration  in  the  forms  of  tables  and  graphs  that  are 
designed  to  convey  the  message  that  expenditures  are  indeed  well  spent.  There  was  a sense  that 
presentation  of  various  statistics  is  being  received  well  within  university  administration. 

As  Penn  devotes  larger  sums  to  the  acquisition  of  digital  information,  it  is  careful  to  maintain 
historical  levels  of  print  acquisition.  The  library’s  goal  is  find  and  keep  a proper  balance  among 
formats.  Thanks  to  steady  increases  in  its  endowment  for  information,  the  library  has  been  able 
to  sustain  higher  levels  of  expenditure  for  networked  resources  without  detriment  to  its  print 
collections. 
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Information  about  the  size  of  electronic  collection  and  use  from  the  collection  are  featured 
prominently  in  the  Library  Facts  documents.  For  instance,  under  the  Digital  Library  headings  the 
following  are  key  items  reported  in  the  99/00  document: 

• Number  of  library  web  pages  accessed  (96/97  to  99/00); 

• Number  of  searches  in  the  OPACs  and  licensed  databases  (96/97  to  99/00); 

• Number  of  journal  article  indexes  and  full-text  files  (96/97  to  99/00); 

• Number  of  electronic  journals; 

• Number  of  records  in  locally  mounted  databases;  and 

• Number  of  locally  digitized  and  accessible  images. 

The  pamphlet  also  reports  the  portion  of  electronic  information  purchase  out  of  total  materials 
expenditure. 

In  terms  of  specific  journal  renewal  and  cancellation  situations,  usage  information  is 
reviewed  by  the  selectors  and  the  Electronic  Acquisitions  Librarian.  However,  low  usage  does 
not  always  lead  to  cancellation.  There  are  other  factors,  political  and  pragmatic,  that  must  be 
considered  in  making  those  decisions. 

Web  statistics  inform  the  Web  Manager  and  other  systems  specialists  to  look  at  better 
webpage  designs  that  promote  easy  access  to  the  pages.  One  of  the  things  that  the  library  does  is 
to  have  an  area  in  the  library  homepage  that  is  devoted  to  the  updates  on  new  electronic 
resources.  The  number  of  access  to  the  links  provides  some  indication  of  user  interests  to  those 
materials. 


Overall,  U Penn  library  is  a leader  in  data  collection/analysis  and  delivering  measures  of 
electronic  resources  and  services.  This  may  be  attributed  to  its  unique  environment  in  terms  of  its 
funding  structure,  the  library  director's  leadership  in  the  area  of  assessment,  and  a unique  pool  of 
human  resources  dedicated  to  the  process.  Having  a fulltime  staff  member  who  acts  as  a data 
advocate  in  the  library  is  also  an  important  factor. 

Other  research  libraries  can  benefit  by  examining  U Penn  Library's  reports  included  in  this 
document  and  available  on  its  website.  However,  the  library's  strength  seems  to  be  embedded  in 
the  organization's  focus  on  evaluation-based  planning  and  execution,  encouraging  a culture  of 
evaluation,  and  committing  resources  to  produce  various  analyses  and  reports. 


Founded  in  1701,  Yale  University  is  a premiere  Ivy  League  institution  located  in  New  Haven, 
CT.  In  addition  to  its  strong  undergraduate  programs  (Yale  College),  the  university  has  many 
world-renowned  professional  programs  in  such  areas  as  law,  medicine,  theology,  business  and 
architecture.  Key  statistics  about  the  library  are  summarized  in  Table  14. 


5)  Summary 


Yale  University  Library 


1)  Overview 


ERIC 
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Table  14.  Yale  Library  Facts 


1997 


1998 


1999 


Total  Expenditure  (in  mil.) 

Materials  Expenditure  (in  mil.) 

Electronic  Resources  Expenditure  (in  mil.) 

Professional  Staff 

Support  Staff 

Total  Students 

Graduate  Students 

Faculty 


39.2 

14.8 

0.9 

201 

311 

10,763 

5,451 

1,792 


42.1 

16.4 

0.9 

212 

316 

10,663 

5,348 

1,792 


42.8 

17.7 

1.0 

217 

345 

10,832 

5,451 

1,825 


2)  Overall  Approach/Philosophy  Toward  Data  Collection  and  Use 

Yale  Library's  approach  to  collecting  and  using  data  needs  to  be  understood  in  the  context 
that  it  is  extremely  committed  to  and  also  capable  of  purchasing  high  quality  information 
resources  in  a wide  range  of  areas  to  its  users.  Individual  subject  specialists  make  purchasing 
decisions  on  both  print  and  electronic  resources  in  their  given  fields  of  interests.  There  are 
currently  about  50  subject  specialists  in  the  library.  Purchasing  of  significant  resources  spanning 
multiple  disciplines  (for  electronic  materials)  is  coordinated  by  the  Electronic  Publishing  and 
Collections  Specialist.  Data  are  sought  to  assist  decision  making  for  the  selectors,  Associate 
University  Library  for  Collection  Development,  instructional  librarians  and  the  Electronic 
Collections  team. 

3)  Data  Collection  Activities 

The  Electronic  Publishing  and  Collections  Specialist  is  in  charge  of  collecting  raw  data  from 
vendors  and  selectors  and  making  them  available  to  internal  staff  members  that  include  subject 
specialists  and  Associate  University  Librarian  for  Collection  Development.  The  responsibility  for 
data  collection  from  vendors  is  gradually  moving  to  the  Digital  Collections  Specialist  who  reports 
to  the  Electronic  Publishing  and  Collections  Specialist. 

Information  is  collected  from  password  protected  vendor  statistics  websites,  email 
attachments  from  the  vendors,  and  print  reports.  The  Electronic  Publishing  and  Collections 
Specialist  has  an  assistant  working  for  her  to  enter  raw  data  from  various  formats  to  html  format. 
The  html  files  are  centrally  managed  in  the  staff  web  area  (see  Appendix  F for  the  image  of  the 
first  screen)  where  links  are  provided  to  not  only  the  current  statistics  but  also  past  statistics. 

Some  of  the  data  goes  back  to  1992  (for  Lexis-Nexis  telnet  product).  There  is  also  a link  to  usage 
statistics  for  retired  electronic  resources. 

According  to  the  assistant,  it  takes  about  3 hours  a week  to  enter  data  into  intermediary 
spreadsheet  (Excel)  PDF  files  and  to  update  the  statistics  page.  For  each  available  vendor 
statistics  report,  there  is  a brief  paragraph  explaining  the  format  of  the  provided  statistics, 
frequency  of  report,  and  any  substantive  changes  in  the  statistics.  Many  statistics  are 
accompanied  by  Excel-generated  graphs  showing  usage  trends.  In  addition  to  the  3 hours  a week 
invested  by  the  assistant,  the  Electronic  Publishing  and  Collections  Specialist  estimates  that  she 
spends  about  2 hours  per  week  on  average  retrieving  and  refining  statistics.  In  addition,  she 
spends  a minimal  amount  of  time  contacting  vendors  to  provide  data. 

Another  major  information  gathering  activity  at  Yale  library  takes  place  at  its  medical  library. 
There,  the  Informatics  Librarian  is  a central  contact  person  for  statistics  on  electronic  materials. 
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At  the  medical  library,  many  full-text  journals  are  locally  mounted  through  OVID  (a  database 
aggregator)  interface  as  well  as  a number  of  databases.  Statistics  are  gathered  on  the  number  of 
sessions  to  each  database  in  the  OVID  offering.  Mode  of  access  (Web,  telnet  or  Z39.50)  is  also 
captured  and  it  shows,  not  surprisingly,  that  many  people  switched  from  character-based  telnet 
access  to  the  resources  to  the  Web  interface. 

Other  kinds  of  information  collected  at  the  medical  library  include  number  of  electronic 
books  consulted,  number  of  electronic  reference  queries,  webpage  access,  and  vendor  supplied 
usage  statistics.  Staff  expressed  some  concern  that  while  a large  amount  of  information  is  being 
collected,  the  library  may  need  to  take  additional  steps  to  use  it  effectively  in  decision  making. 

4)  Data  Use  and  Main  Users 

Since  so  much  effort  has  gone  into  acquiring  statistics  from  the  vendors  and  converting  them 
to  uniformly  accessible  forms,  the  data  are  taken  quite  seriously.  Usage  statistics  are  a strong 
element  in  the  review  of  cost  effectiveness  of  purchased  databases.  However,  usage  is  not 
currently  the  only  factor  in  electronic  resource  subscription  cancellations:  usage  statistics  are 
weighed  with  and  against  other  factors  as  well.  Usage  statistics  are  passed  along  to  the  subject 
specialist.  Staff  indicated  that  Yale  library,  as  many  other  research  libraries,  is  having  problems 
with  vendors  who  do  not  provide  any  statistics  at  all.  In  that  case,  it  is  difficult  to  understand  use 
and  take  appropriate  actions  to  improve  service. 

One  of  the  specific  areas  where  data  are  used  very  effectively  is  deciding  the  number  of 
simultaneous  users  that  are  licensed  for  access  to  a given  database,  where  simultaneous  user 
(S/U)  is  the  payment  model.  Tum-away  figures  from  vendor  usage  statistics  are  closely 
examined  and  related  to  the  number  of  user  complaints. 

Statistical  usage  data  are  distributed,  for  the  most  part,  internally.  For  the  last  several  years, 
the  first  issue  of  Nota_Bene,  a library  newsletter  published  three  times  a year,  contained  an  annual 
report  from  the  university  librarian.  It  is  a narrative  summary  of  major  library  events  and 
accomplishments  and  not  intended  as  a full  report  containing  detailed  information  about  the  use 
of  library  collections  and  services  including  electronic  materials.  In  the  spring  1999  issue, 
however,  there  was  a report  by  the  Electronic  Publishing  and  Collections  Specialist  about  the 
increase  of  electronic  resource  use  with  JSTOR  and  Eureka  as  examples 
(http://www.librarv.vale.edu/NotaBene/nbxiii2/arcxiii2.htm).  In  the  medical  library,  most  of 
statistics  that  are  mentioned  as  collected  are  reported  in  the  library's  annual  report. 

5)  Summary 

Among  many  of  its  strengths  in  the  area  of  collection  of  data  regarding  electronic  resources 
two  points  are  worth  mentioning.  The  first  one  is  that  there  is  a staff  member  who  maintains  an 
on-going,  clear  communication  with  database  vendors.  This  has  not  only  benefited  Yale  Library 
but  also  other  research  libraries  in  some  instances.  The  second  strength  is  their  information 
organization  skill  which  transforms  a range  of  data  into  separate  information  products  available  to 
involved  parties  for  reference  and  decision  making. 
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The  New  York  Public  Library,  the  Research  Libraries 


1)  Overview 

The  New  York  Public  Library  is  the  only  facility  of  its  kind,  with  both  world-acclaimed 
research  centers  and  a large  network  of  neighborhood  branch  libraries,  all  of  which  may  be  used 
by  the  public,  free  of  charge.  The  Research  Libraries  consist  of  four  centers:  the  New  York 
Public  Library  for  the  Performing  Arts;  the  Schomburg  Center  for  Research  in  Black  Culture;  the 
Humanities  and  Social  Sciences  Library;  and  the  Science,  Industry  and  Business  Library.  Among 
these,  the  Humanities  and  Social  Sciences  Library  (HSSL);  and  the  Science,  Industry  and 
Business  Library  (SIBL)  are  the  principal  users  of  electronic  information.  Key  statistics  about  the 
library  are  summarized  in  Table  15. 

Table  15.  NYPL  (The  Research  Libraries)  Facts 


1997 

1998 

1999 

Total  Expenditure  (in  mil.) 

43.0 

43.7 

44.8 

Materials  Expenditure  (in  mil.) 

9.5 

10.4 

11.0 

E-Resources  expenditure  (in  mil) 

0.9 

1.2 

1.3 

Professional  Staff 

253 

259 

n/a 

Support  Staff 

373 

368 

n/a 

n/a  : not  available 


2)  Overall  Approach/Philosophy  Toward  Data  Collection  and  Use 

One  important  fact  that  distinguishes  NYPL  as  a research  library  from  other  ARL  libraries  is 
that  it  does  not  have  a captive  pool  of  users.  While  the  potential  service  pool  is  almost  limitless, 
it  also  makes  it  difficult  for  the  staff  to  develop  focused  service  programs  for  clients. 

As  far  as  the  electronic  collection  is  concerned,  access  to  most  licensed  databases  is  provided 
on-site.  There  are  about  a dozen  licensed  databases  that  are  available  to  registered  users  for  home 
use  through  shared  subscription  with  branch  libraries  (85  branches  as  of  8/30/00).  However, 
these  databases  are  geared  toward  general  public  use  rather  than  for  research  purposes. 

Since  NYPL  receives  part  of  its  funding  from  the  city  and  state  governments,  there  is  a 
specific  reporting  requirement  in  regard  to  these  expenditures.  For  example,  the  state  government 
requires  that  the  library  report  the  number  of  electronic  database  subscriptions  and  the  overall 
cost.  More  detailed  requirements  do  not  appear  to  have  occurred  as  yet. 

3)  Data  Collection  Activities 

Vendor  statistics  for  licensed  materials  are  gathered  by  the  Research  Library’s  Chief  Librarian 
for  Acquisitions,  whose  division  is  responsible  for  the  materials  acquired  for  all  four  Research 
Libraries.  Data  also  flows  from  electronic  resources  specialists  in  the  various  units  of  the 
Research  Libraries  to  the  Chief  Librarian.  There  is  an  Electronic  Resources  Group  (ERG)  that 
reviews  purchasing  requests  from  selectors  and  coordinates  purchasing  decisions  among  Research 
Libraries.  Purchasing  is  centralized  across  the  Research  Libraries. 

Every  month,  vendor  supplied  use  statistics  are  entered  into  a monthly  statistical  usage  report 
by  each  database  in  the  licensed  package.  The  most  representative  measure  of  use  statistics  is 
assigned  to  the  database  to  calculate  the  unit  cost  per  each  use  instance.  For  many  databases,  the 
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vendor-supplied  statistic  is  number  of  searches.  Number  of  records  accessed,  items  viewed,  and 
number  of  requests  are  examples  of  additional  metrics  used.  There  is  no  information  about  what 
these  measures  mean  as  they  are  taken  directly  from  vendor  reports  and  the  vendors  do  not 
provide  definitions  in  the  reports. 

4)  Data  Use  and  Main  Users 

Database  use  is  scrutinized  for  being  cost-effective.  The  main  objective  is  to  spot  low-use 
databases  and  consider  decisions  regarding  product  placement  and  promotion  of  electronic 
resources.  Some  of  the  ways  to  promote  a particular  database  or  resource  is  to  feature  it  on  the 
web  site  or  in  public  training  classes. 

SBIL  (the  Science,  Industry  and  Business  Library)  offers  free  user  instruction  courses  on  a 
wide  range  of  topics  and  interests.  These  sessions  feature  a variety  of  electronic  resources.  SLBL 
averages  about  900  training  class  participants  per  month.  Other  Research  Libraries  will  be 
developing  structured  training  programs  for  users  in  the  coming  months. 

Monthly  database  use  statistics,  containing  available  use  counts  for  each  vendor  along  with 
annual  cost  and  per  use  cost,  are  circulated  among  ERG  members  and  shared  within  their  own 
units.  A member  of  the  ERG  is  the  Deputy  Director  and  through  that  person,  a summarized 
version  of  the  statistics  is  reported  to  the  Research  Libraries  Management  Team  headed  by  the 
Director  of  the  Research  Libraries. 

As  far  as  an  official  channel  of  information  dissemination  regarding  these  data,  the  closest 
thing  appears  to  be  the  annual  report.  The  report  describes  the  overall  activity  and  summary  facts 
(e.g.,  major  contribution  and  financial  statements)  of  the  NYPL  and  is  not  limited  to  the  Research 
Libraries.  In  the  annual  report  (http://www.nypl.org/admin/pro/ar/annualrpt.htmIL  the  following 
four  items  are  summarized  under  the  cyberfacts  heading  in  Facts  & Figures  section: 

• Electronic  visits  to  NYPL  website  annually; 

• Countries  accessing  website; 

• Computers  for  public  use;  and 

• Electronic  databases  for  public  use. 

These  numbers  are  for  the  entire  NYPL,  including  both  the  Branch  Libraries  and  the  Research 
Libraries. 

5)  Summary 

The  most  noticeable  factor  at  NYPL  is  its  ability  to  relate  the  collected  data  on  electronic 
resources  to  its  instructional  programs.  User  education  is  a big  topic  with  which  many  research 
libraries  are  concerned,  and  NYPL's  expertise  in  offering  high  quality  programs  can  benefit  other 
research  libraries.  A second  factor  to  stress  is  NYPL's  on-going  cost/benefit  analysis  of  its 
electronic  resources.  The  decision  process  for  this  analysis  may  be  of  interest  to  other  libraries. 


Summary  of  Observations 

Libraries  are  under  different  operating  environments  and  have  different  needs  in  terms  of  data 
for  electronic  resources.  The  environment  and  attitudes  towards  assessment  are  being  shaped  by 
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the  institution's  involvement  with  the  library  operation  and  the  library's  top  management  attitude 
toward  evaluation  efforts.  Support  for  the  institution's  involvement  can  be  manifested  in  various 
forms  such  as  library  funding  structure,  provision  of  staff,  and  use  of  various  data  to  support 
decision-making. 

Libraries  are  having  a serious  problem  managing  information  on  the  use  of  electronic 
resources  and  services,  particularly  with  regard  to  licensed  vendor  materials.  Information  sources 
often  reside  under  vendor  control.  Although  most  information  delivery  is  carried  out  on  via  the 
Web,  users  have  to  go  through  different  interfaces  for  different  types  of  resources.  Accordingly, 
usage  statistics  are  distributed  among  typically  several  dozen  database  vendors  and  consortia. 

Due  to  a lack  of  standardized  reporting  practices,  usage  reports  are  hard  to  consolidate,  or  it  takes 
an  enormous  amount  of  effort  to  collect  such  data.  Communication  between  database  providers 
and  the  research  library  community  is  sorely  needed  to  address  the  problem. 

While  libraries  are  making  progress  in  many  areas  of  measurement  of  electronic  resources, 
libraries  have  yet  to  succeed  in  producing  a coherent  picture  about  the  utilization  of  electronic 
information.  While  there  certainly  are  a range  of  issues  that  have  yet  to  be  addressed  here,  some 
steps  can  be  taken  now  by  ARL  libraries.  We  will  discuss  preliminary  suggestions  that  can  help 
research  libraries  move  in  that  direction  in  the  next  chapter. 
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PART  FOUR:  ISSUES,  SHORT-TERM  RECOMMENDATIONS, 

AND  NEXT  STEPS 

A key  question  addressed  by  this  study  is:  how  can  ARL  libraries  become  more  successful  in 
describing  networked  information  resources  and  services  at  their  libraries?  As  the  Phase  I report 
indicates,  it  is  difficult  to  delineate  specific  factors  leading  to  success  as  yet  — indeed,  it  may  be 
too  early  to  offer  “best  practices”  given  the  state  of  knowledge  in  this  area. 

To  a large  degree  ARL  libraries  have  just  begun  looking  at  this  topic  seriously  and  formal 
assessment  activities  in  this  area  differ  among  ARL  libraries.  Clearly,  there  are  a number  of 
libraries  that  are  working  in  this  area  and  gaining  experience  with  some  specific  procedures  such 
as  cost  analysis  and  collection  development.  But  as  a group,  it  is  difficult  to  point  to  specific  best 
practices  among  participating  libraries  and  offer  a coherent  picture  of  issues,  strategies,  and 
specific  techniques  related  to  producing  statistics  and  performance  measures  to  describe 
networked  services. 

1.  Selected  Issues 

The  findings  from  Phase  I (see  previous  chapter)  suggest  that  there  are  a number  of  key 
issues  that  could  benefit  from  additional  discussion  and  research.  This  section  briefly  identifies 
and  describes  some  of  these  issues. 

The  Complexity  of  This  Topic 

As  the  study  team  and  the  participating  libraries  continue  through  this  project,  it  appears  that 
the  issues  and  possible  approaches  for  developing  statistics  and  measures  for  the  networked 
environment  may  be  more,  rather  than  less,  complicated  than  anticipated  originally.  Moreover, 
the  degree  of  this  complexity  is  only  beginning  to  be  understood. 

One  project  participant  scolded  a member  of  the  study  team:  why  haven't  we  already 
produced  draft  performance  measures  that  all  ARL  libraries  could  use  to  describe  journal  use 
from  vendor-supplied  databases.  When  the  study  team  member  explained  that  there  were  still  a 
number  of  issues  and  problems  necessary  to  address  before  it  is  possible  to  develop  such 
measures,  the  participant  replied,  “I  don’t  care  about  the  problems,  just  give  me  the  measure.” 
Thus,  it  is  necessary  to  better  understand  the  range  of  the  complexities  affecting  the  development 
of  statistics  and  performance  measures  by  both  the  library  and  the  vendor  community. 

Given  the  complexity  of  the  topic,  there  is  a need  to  better  educate  participating  libraries, 
university  administrators,  vendors,  and  others  as  to  the  nature  of  these  complexities  and  how  they 
affect  developing  statistics  and  performance  measures.  These  complexities  include  factors  related 
to  both  data  collection  and  analysis  as  well  as  organizational/institutional  factors. 


Diverse  Context  for  Developing  Statistics  and  Performance  Measures 

There  are  a range  of  unique  and  situational  factors  at  the  various  participating  libraries  that 
affect  the: 

(1)  Library’s  need  for  statistics  and  performance  measures; 

(2)  Degree  to  which  the  library  is  willing  to  commit  resources  to  produce  such  statistics 
and  performance  measures; 
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(3)  Range  and  type  of  databases  being  used  by  the  library; 

(4)  Library’s  staff  relationships  with  the  vendors  of  these  databases;  and, 

(5)  Specific  type  of  statistics  and  performance  measures  that  would  be  useful  in  a 
particular  library  setting. 

In  some  libraries,  there  is  less  interest  from  university  administration  as  to  the  benefit,  impact, 
and  use  of  electronic  resources  with  resulting  “less  pressure”  to  produce  descriptive  data.  Other 
libraries  have  significant  cost  and  administrative  pressure  to  produce  such  data.  Thus,  the  nature 
of  the  university  context  in  which  a library  operates  has  a significant  impact  on  the  development 
of  statistics  and  performance  measures  to  describe  their  networked  services  and  resources. 

This  diverse  context  suggests  that  while  there  may  be  a set  of  “core”  statistics  and 
performance  measures  that  would  be  of  use  to  all  libraries,  there  is  also  likely  to  be  unique 
statistics  and  measures  that  will  be  of  greater  interest  to  some  rather  than  others.  A better 
understanding  of  local  factors  that  affect  the  need  for  specific  types  of  statistics  and  performance 
measures  will  require  additional  attention.  Models  that  can  help  describe  these  local  situations 
may  also  be  useful  in  the  selection  of  statistics  and  performance  measures. 

ARL  Library  Responsibilities  and  Level  of  Effort 

The  results  from  Phase  I suggest  a wide  range  of  self-induced  responsibilities  and  level  of 
effort  on  the  part  of  ARL  libraries  for  data  collection,  analysis,  and  reporting.  In  the  Scottsdale 
meeting,  more  than  half  of  participants  indicated  that  they  had  no  data  collection  activities  in  the 
area  of  electronic  services  and  resources,  while  others  reported  some  considerable  effort. 

To  some  degree,  the  amount  of  responsibility  and  level  of  effort  that  the  library  will  commit 
to  data  collection  and  analysis  of  networked  services  and  resources  depends  on  their  context,  the 
perceived  importance  of  having  such  data,  and  how  such  data  might  be  used  effectively  in  the 
local  setting.  Yet  to  be  understood  is  what  constitutes  “reasonable”  levels  of  effort  on  the  part  of 
the  library  to  collect  and  produce  such  data  at  a given  library  or  at  libraries  in  general.  Some 
libraries  may  need  only  to  commit  minimal  resources  to  obtain  the  data  that  they  need,  whereas 
others  may  have  to  commit  significant  resources  in  order  to  obtain  the  data  required  for  their 
decision  making  processes.  Again,  models  that  describe  and  relate  local  level  of  efforts  to  uses 
and  applications  of  statistics  and  performance  measures  may  be  useful. 

Focus  on  Non-Vendor-Based  Data 

Phase  I of  the  study  suggests  that  a number  of  libraries  are  not  well  prepared  to  collect, 
analyze,  and  report  data  related  to  networked  services  and  resources  — regardless  of  whether  the 
data  comes  from  vendors  or  from  internal  library  activities.  Further,  there  are  a number  of  data 
elements  related  to  networked  and  electronic  services  that  are  not  dependent  on  data  being 
supplied  by  vendors.  For  example,  statistics  related  to  counting  and  describing  users  of  electronic 
reference  (web-based  or  email  based)  are  not  dependent  on  obtaining  data  from  vendors.  Yet, 
there  is  some  sense  to  focus  on  the  vendor-based  statistics  rather  than  other  types  of  networked 
and  electronic  statistics. 

While  the  database  statistics  supplied  by  vendors  are  essential,  there  may  be  other  types  of 
electronic  data  that  libraries  might  try  to  collect  and  analyze.  As  suggested  in  the  findings  section 
of  the  study  and  in  selected  appendices,  there  are  statistics  and  measures  being  collected  that 
describe  activities  other  than  database  use,  users,  and  uses.  In  the  short  term,  libraries  may  want 
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to  continue  their  efforts  in  these  areas  until  there  is  some  more  general  agreement  with  vendors  on 
obtaining  and  using  statistics  from  those  vendors. 

Coordination  Among  Libraries  and  Library  Organizations 

In  previous  work  conducted  by  the  authors,  a number  of  vendors  noted  that  different  libraries, 
different  library  professional  organizations,  and  different  national  organizations  have  different 
needs  and  agenda  regarding  the  development  of  statistics  and  performance  measures  to  describe 
networked  services  and  resources.  Minimally,  there  are  the  following  players  in  this  discussion: 

• Individual  libraries  (of  all  types).  Particular  libraries  may  have  specific  needs  for  collecting 
data,  analyzing  data,  and  the  manner  in  which  data  are  reported  to  them,  thus,  they  contact  the 
vendors  directly  to  express  their  needs. 

• Professional  associations.  Various  groups  in  the  professional  associations  regularly  meet  to 
discuss  data  needs  from  vendors  and  how  best  to  obtain  that  data. 

• National  Commission  on  Libraries  and  Information  Science  (NCLIS).  The  Commission  is 
currently  engaged  in  a project  to  facilitate  public  librarians,  state  libraries,  and  selected 
consortia  to  work  with  vendors  to  describe  the  types  of  data  that  should  be  made  available 
and  how  it  should  be  reported  to  libraries. 

• Funded  Research  Projects.  Currently  the  authors  are  funded  by  a group  of  24  ARL  libraries 
to  work  with  vendors  to  develop  statistics  and  performance  measures  (among  other 
objectives).  The  authors  are  also  funded  by  a grant  from  the  U.S.  Institute  of  Museum  and 
Library  Services  (IMLS)  to  develop  national  models  and  techniques  for  standardized  data 
collection  of  networked  information  services  and  resources  for  public  libraries.  There  are 
likely  to  be  other  such  funded  research  projects. 

• ICOLC.  This  group  has  met  for  a number  of  years  and  currently  has  guidelines  regarding  the 
definitions  and  data  collection  strategies  for  databases  (see  Appendix  C). 

• Council  on  Library  and  Information  Resources  (CLIR).  This  organization  has  contracted 
with  Judy  Luther  to  write  a white  paper  on  the  E-Joumal  Usage  Statistics 
<http://www.clir.org/pubs/reports/pub94/contents.html.  CLIR  can  be  expected  to  continue  its 
work  in  this  area. 

• National  Information  Standards  Organization  (NISO).  The  international  standard  for  library 
statistics  is  currently  under  review  and  NISO  is  leading  a process  in  the  United  States  to 
develop  revised  standards  for  library  statistics  including  database  related  statistics. 

There  may  be  other  key  groups  currently  involved  in  developing  database  statistics  and 
performance  measures.  Nonetheless,  it  is  imperative  that  the  efforts  described  here  are 
coordinated  such  that  vendors  are  not  besieged  by  multiple  individual  s/organizations  with 
multiple  (and  possibly  confounding)  objectives. 

Summary 

This  section  is  not  intended  to  be  a comprehensive  listing  of  issues  that  the  study  team 
identified  as  part  of  Phase  I.  These  issues  do,  however,  suggest  that  there  is  a wide  range  of 
opinions  and  approaches  regarding  the  topic  with  relatively  little  agreement  as  to  specific 
strategies  (both  at  individual  libraries  and  among  various  organizations)  that  might  be  best 
pursued  while  the  project  continues. 
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2.  Short-term  Recommendations 

This  section  offers  specific  strategies  and  suggestions  that  may  be  useful  for  participating 
libraries  to  consider  as  the  project  continues.  We  expect  these  suggestions  will  become  more 
formalized  as  the  study  continues  and  as  we  witness  improvements  in  library  practices.  Some  of 
our  suggestions  address  issues  that  may  not  seem  directly  related  with  measuring  electronic 
services  and  resources.  For  example,  measurement  of  library  and  user  activities  related  to 
electronic  information  services  and  resources  should  not  be  viewed  in  and  of  itself.  Rather  the 
process  has  to  be  incorporated  into  the  overall  library  assessment  effort.  We  believe  that  the 
research  library  community  already  recognizes  this  issue  and  is  working  toward  a more  holistic 
approach  to  evaluation. 

As  previously  noted,  we  recognize  that  libraries  operate  in  different  contexts.  We  also 
recognize  that  libraries  have  different  needs  to  fulfill  and  varying  amounts  of  resources  to  tap. 
Nonetheless,  we  believe  that  these  short-term  recommendations  and  suggestions  will  be  useful 
regardless  of  the  specific  types  of  statistics  and  performance  measures  that  ultimately  are 
recommended  by  the  study. 

Libraries  Should  Cultivate  a Culture  of  Assessment 

There  has  been  some  work  done  on  “culture  of  assessment.”  Amos  Lakos  (1999,  p.  5)  defines 
culture  of  assessment  in  libraries  as: 

The  attitudinal  and  institutional  changes  that  have  to  occur  in  order  for  library  staff  to  be  able 
to  work  in  an  environment  where  decisions  are  based  on  facts,  research  and  analysis,  and 
services  are  planned  and  delivered  in  order  to  maximize  positive  outcomes  and  impacts  for 
the  library  clients.  Culture  of  assessment  is  an  integral  part  of  the  progress  of  change  and  the 
creation  of  a customer-centered  culture. 

Such  a culture  is  essential  if  the  library  is  to  move  successfully  into  assessment  of  networked 
information  services  and  resources. 

Assessment  of  library  performance  of  networked  information  resources  cannot  be  separated 
from  the  overall  assessment  of  the  library's  roles  and  the  extent  of  fulfillment  of  those  to  the  user 
community.  It  is  a very  encouraging  sign  that  many  libraries  expressed  a desire  to  investigate  not 
only  their  performance  using  electronic  resources  but  also  how  such  performance  is  related  to  the 
overall  library  mission.  One  library  director  during  the  site  visit  commented,  "Everything  is 
related.  So  we  need  to  concentrate  on  non-electronic  information  too.  E-data  isn't  important  just 
by  itself.  It  is  part  of  a total  picture  that  is  being  examined." 

Library  directors  and  other  staff  will  need  to  foster  recognition  of  the  importance  of 
assessment  in  his/her  library.  Although  this  issue  has  been  discussed  for  several  decades  in  the 
library  community,  we  repeat  it  here  again:  unplanned,  unsystematic  approaches  to  evaluation 
will  satisfy  only  short-term  needs.  A culture  of  assessment  requires  making  assessment  a library 
priority  as  well  as  careful  planning.  If  a library  is  considered  performing  well,  it  has  to  be 
demonstrated  by  supporting  data.  In  some  cases,  trust  among  the  user  community  and  the 
institution  can  be  regarded  as  a sign  of  library  success,  not  as  an  excuse  for  not  engaging  in  active 
assessment  activities.  Even  then,  the  library  can  perform  better  if  it  operates  on  the  premise  that 
assessment  can  drive  better  performance. 
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Libraries  Need  to  Plan  and  Implement  Organizational  Structures 
That  Support  a Changing  Statistical  Environment. 

Rapid  penetration  of  electronic  materials  and  services  into  the  research  community  and 
changing  user  demands  of  information  prompt  research  libraries  to  reconsider  their  current 
organizational  structure  to  assist  in  collecting  data  for  performance  measurement.  The  size  and 
specific  forms  of  needed  organizational  structure  depend  on  the  library's  needs  and  resources. 
However,  there  are  several  guidelines  to  consider. 

One  of  the  key  issues  regarding  organizational  structure  is  that  it  is  critical  to  have  a library 
office  specifically  involved  and  responsible  for  evaluation  and  assessment.  This  is  a difficult 
issue  for  some  libraries  because  of  a shortage  of  library  technical  staff.  In  some  libraries,  a 
library  system's  resource  pool  is  stretched  simply  to  keep  up  with  the  day-to-day  operations. 
Nonetheless,  the  systems  (or  other)  department  needs  to  run  programs  or  even  write  special 
programs  to  capture  or  process  data  to  produce  a range  of  statistics  and  measures.  Reliable  and 
high  quality  data  cannot  be  obtained  unless  data  needs  are  identified  in  advance  and  integrated 
into  the  organizational  information  systems  large  and  small.  While  off-the-shelf  software 
solutions  are  essential  and  have  their  place,  libraries  may  need  to  develop  their  own  data 
gathering  and  analysis  infrastructure  to  suit  their  special  needs. 

The  study  team  conducted  a quick  poll  to  describe  the  organizational  structure  that  the  project 
participants  currently  have  to  coordinate  the  collection  and  analysis  of  data  on  electronic  resource 
use.  Fourteen  libraries  responded  to  the  question.  Among  them,  only  six  libraries  said  someone 
in  the  systems  or  a technical  department  is  represented  in  the  data  collection  and  analysis  process. 
The  responses  from  the  fourteen  libraries  are  summarized  in  Appendix  G. 

Also  apparent  in  the  responses  was  an  apparent  lack  of  ownership  in  the  process,  in  the 
reported  organizational  structures,  and  the  limited  scope  of  the  structures’  roles.  Too  often,  these 
groups  deal  with  only  one  aspect  of  data  collection,  e.g.,  licensed  materials,  and  it  is  not  clear 
who  is  responsible  for  data  analysis  and  presentation  in  other  areas.  It  may  be  too  limiting  to 
have  the  committee  or  team  simply  collect  raw  information  and  pass  among  the  group  and  to 
other  parties  in  the  library.  More  useful  is  a team  or  specific  individual  being  charged  with  the 
whole  data  collection/analysis/reporting  process  throughout  the  collection,  processing,  analysis 
and  presentation  of  evaluation  information. 

A number  of  participants  reported  that  the  organizational  structures  for  assessment  were 
designed  to  facilitate  purchasing  of  electronic  resources.  However,  the  scope  of  those  groups 
may  well  need  to  be  expanded  to  cover  other  important  issues  such  as  changing  user  behaviors, 
education  of  library  staff  and  faculty,  and  considering  demands  and  the  dynamics  between  print 
and  electronic  materials.  Again,  we  notice  that  several  libraries  are  already  moving  in  this 
direction. 


Libraries  Should  Have  Clear  Statements  about  Their  Assessment  Goals 
in  Key  Documents  Such  as  Strategic  Planning. 

Some  libraries  are  not  clear  about  why  they  are  engaged  in  the  measurement  activity.  Many 
were  able  to  articulate  the  immediate  needs  such  as  to  make  purchase  decisions  and  to  justify 
budget  decisions.  But  some  library  organizations  seem  to  have  difficulty  in  being  able  to  link  the 
measurement  effort  to  the  overall  goals  of  the  library  and  the  larger  institution. 
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Strategic  planning  is  a useful  tool  to  answer  questions  such  as  "what  are  the  objectives  of  data 
collection  and  analysis?"  By  describing  specific  measurement  efforts  for  each  of  the  library's 
goals  and  implementation  plan,  libraries  can  relate  specific  data  collection  and  analysis  activities 
to  the  overall  goal  of  the  library. 

Another  important  aspect  of  strategic  planning  is  including  measurement  activity  itself  as  a 
major  goal  or  priority.  The  University  of  Pennsylvania  Library,  for  example,  promulgates 
developing  new  ways  of  measuring  library  effectiveness  as  a key  goal  in  its  1996-2000  strategic 
planning.  Only  a few  other  participating  libraries  mentioned  assessment  and  data  collection 
related  to  networked  services  in  their  strategic  plan. 

It  is  important  that  measurement  areas  and  goals  are  clearly  identified  in  the  strategic 
planning  document.  Otherwise,  it  is  likely  that  they  may  be  ignored.  Having  a well-planned 
initiative  in  a specific  area  with  measurement  goals  and  objectives  will  facilitate  the  planning 
process.  By  doing  so,  processes  and  experiences  gained  can  be  used  in  other  areas  of  library 
measurement. 


Importance  of  a Data  Advocate 

Having  an  official  data  advocate  in  the  library  to  measure  both  traditional  and  electronic 
information  resources  and  services  can  enhance  the  success  of  measurement  efforts.  At  most 
ARL  libraries,  there  is  a staff  member  who  has  responsibility  for  collecting  statistics  for  internal 
use  and  for  outside  organizations,  most  often  ARL  statistics.  This  person  may  also  be  able  to 
serve  as  a data  advocate  within  the  library. 

Given  the  complexity  of  measurement  and  evaluation  in  today’s  library,  eventually  the  role  or 
position  may  need  to  become  a full-time  position  or  positions  depending  on  the  library's  resources 
and  expectations.  The  person  works  closely  with  library  management  in  terms  of  information 
needs  for  various  purposes.  The  official  title  can  vary.  But  it  is  important  that  the  person 
becomes  visible  in  the  library  and  be  deeply  involved  in  the  entire  process  of  data  collection, 
analysis,  and  presentation.  An  ideal  situation  may  be  one  where  the  data  advocate  coordinates  the 
organization's  overall  assessment  activities  including  fulfilling  outside  reporting  requirements  and 
working  directly  with  vendors. 

This  kind  of  position  can  assist  libraries  to  coordinate  data  and  assessment  into  the  overall 
organization's  activities  not  only  to  meet  the  needs  of  specific  functions  or  departments  but  to 
meet  external  requirements  as  well.  Also  since  most  data  are  funneled  through  this  person,  there 
may  be  increased  reliability  and  validity  of  the  data  for  both  the  data  and  the  measures.  This 
person  can  function  as  a resource  to  the  library's  measurement  initiatives.  The  desired  skill  sets 
for  data  advocate  include  excellent  communication  skills,  an  ability  to  educate  staff  and  faculty  as 
to  the  importance  of  such  data,  as  well  as  data  analysis,  and  presentation  skills. 

Develop  Action-Oriented  and  Problem-Oriented  Data  Gathering 

Data  should  not  be  collected  simply  because  they  are  available.  A key  criterion  for  what  data 
to  collect  is  the  degree  to  which  that  data  address  specific  problems,  adds  information  to  decision- 
making, or  otherwise  improves  library  services.  For  example,  some  libraries  employ  WebTracker 
or  WebTrends  to  produce  a range  of  log  analyses.  But  if  the  programs  are  run  without  first 
thinking  about  what  data  are  necessary  for  what  problems  and  decision  situations,  it  is  likely  that 
the  data  will  remain  in  the  various  reports  generated  by  the  software.  A number  of  library  staff 
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commented  that  such  software  was  in  use  at  their  library  but  that  the  resultant  reports  were  largely 
ignored. 

A good  approach  is  to  have  clear  objectives  of  why  the  library  needs  certain  data  and  what 
problems  or  decision  situations  should  be  addressed  by  the  data.  A limited  amount  of  data 
gathered  with  clear  intentions  is  much  more  valuable  than  a vast  amount  of  data  with  no  specific 
purpose. 

It  is  not  likely  that  we  will  ever  have  satisfactory  data  for  all  of  our  electronic  resources.  The 
skill  libraries  have  to  develop  is  to  act  and  make  decisions  based  on  “good  enough”  data.  We 
should  look  for  data  that  can  support  "best  assessment"  decision-making.  What  is  important  is 
that  decisions  are  made  and  actions  are  taken  based  on  data  or  its  analysis.  Most  frequently,  data 
about  electronic  resource  usage  are  used  primarily  for  assessing  contracts.  But  the  same  data  can 
be  effectively  used  in  reporting  to  and  communicating  with  the  user  community  and  supporting 
organizations.  Perhaps  most  importantly,  there  seems  room  for  using  the  data  for  service 
assessment  and  improvement. 

Certain  amounts  of  experimentation  seem  inevitable  given  the  fact  that  we  are  in  a very 
different  operating  environment  now  as  opposed  to  ten  years  ago.  While  it  may  be  extremely 
difficult  to  make  sense  of  inconsistent  data  elements  and  formats,  more  effort  is  necessary  to 
direct  our  attention  to  actions  that  can  further  enhance  service  delivery  and  problems  that  can  be 
corrected  using  the  data.  Thus,  libraries  should  carefully  consider,  outline,  and  detail  the  areas 
where  data  collection  and  analysis  is  needed  in  their  libraries. 

Libraries  Need  to  Maintain  Open  and  On-Going  Effective  Communication 
with  Data  Sources  Including  Database  Vendors,  Users,  and  Internal  Staff. 

Clearly,  inconsistent  and  non-standardized  vendor  reports  pose  a serious  problem  to  libraries. 
It  puts  significant  strain  on  the  part  of  libraries  to  collect  and  process  needed  information.  Even 
more  problematic  are  the  vendors  who  do  not  provide  any  information.  On  the  other  hand, 
database  vendors  need  to  know  why  libraries  need  these  statistics  and  how  they  are  going  to  use 
them.  Although  libraries  need  to  speak  with  “one  voice”  as  to  definitions,  report  formats,  etc., 
libraries  also  need  to  have  realistic  expectations  regarding  vendors’  capabilities  in  terms  of 
generating  useful  information. 

There  is  some  concern  among  the  library  community  as  to  the  reasons  why  vendors  may  be 
reluctant  to  provide  usage  statistics.  Some  participants  indicated  a fear  that  if  the  data  were 
provided  the  library  might  then  cancel  services.  With  more  database  vendors  showing  interest  in 
providing  usage  measures,  there  is  considerable  likelihood  that  we  can  make  a good  progress  in 
this  area.  While  a concerted  effort  like  the  E-Metrics  project  will  play  an  important  role  in 
initiating  the  process,  it  is  also  important  for  individual  libraries  to  keep  open  and  effective  on- 
going communications  with  vendors  in  terms  of  service  delivery  and  providing  usage  statistics. 

Effective  communication  regarding  issues  outlined  in  this  report  will  also  have  to  occur 
between  library  and  other  constituencies  including  user  communities,  university  administration, 
and  library  staff.  Understanding  their  needs  for  statistics  and  measures  is  an  important  step 
toward  quality  service  and  improved  performance.  Ongoing  dialogue  with  the  vendor  community 
regarding  statistics  is  essential  for  developing  a mutually  beneficial  data  reporting  system. 

Libraries  Need  to  Develop  Strategies  and  Skills  to  Produce 
Information  Products  for  Organizing  and  Presenting  data. 
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Libraries  need  to  produce  effective  assessment  products  for  networked  services  and  resources 
with  the  resources  they  have  available  currently.  Libraries  have  to  wrestle  with  ways  in  which 
they  can  package  information  regarding  networked  services  and  disseminate  it  effectively.  For 
instance  Yale  and  other  libraries  have  developed  a central  staff  website  dedicated  to  usage 
statistics.  The  University  of  Pennsylvania  Library's  high  quality  brochures  are  another  good 
example. 

At  an  individual  level,  one  of  the  highly  useful  skills  necessary  for  many  librarians  is  the 
ability  to  utilize  data  processing  programs  such  as  Microsoft  Excel  spreadsheet  or  Access 
database  software.  Excel  has  shown  itself  as  the  de  facto  tool  for  organizing  raw  data  and 
performing  necessary  analysis  such  as  charting  graphs  and  summary  tables.  It  is  evident  in  the 
library-generated  sample  reports  that  some  of  the  librarians  have  excellent  skills  using  tools  like 
Excel.  Presentation  of  the  data  affects  and  is  directly  related  to  its  reception  and  use. 

One  activity  libraries  can  do  in  this  area  is  to  provide  training  programs  for  staff  to  use 
advanced  data  analysis  and  presentation  tools.  Another  is  to  promote  and  encourage  high  quality 
information  products  designed  for  audiences  within  and  outside  of  libraries.  Certain  data 
presentation  templates  need  to  be  used  consistently  across  library  publications.  Books  written  by 
information  design  experts  such  as  Edward  Tufte  (1990,  1992,  and  1997)  and  Jakob  Nielson 
(1999)  offer  excellent  guidelines  and  examples. 

In  conclusion,  while  libraries  are  making  early  progress  in  gathering  data  on  electronic 
information  use,  a range  of  improvements  can  occur  for  more  systematic  and  fruitful 
measurement  efforts.  Some  of  the  recommendations  discussed  in  this  section  relate  to  developing 
a culture  of  assessment,  new  organizational  structures  for  evaluation  and  data  collection, 
decision-oriented  data  gathering,  and  encouraging  measurement  to  be  a key  component  in 
strategic  planning.  Other  suggestions  outlined  in  this  section  are  more  practical:  keeping  an  open 
and  ongoing  communication  process  with  vendors  and  users;  having  a dedicated  person 
responsible  for  the  library's  overall  data  needs;  and,  developing  high-quality  information  products 
to  disseminate  library  performance. 


3.  Next  Steps 

Next  steps  in  Phase  Two  of  the  E-Metrics  Project  are  outlined  in  the  original  Project  Proposal 
submitted  to  ARL,  April  27,  2000,  and  will  not  be  repeated  here.  These  steps  will  be  informed  by 
the  findings  and  results  from  Phase  I.  Key  activities  for  Phase  II  include  the  following. 

Work  with  Major  Vendors  to  Standardize  Usage  Reports 

The  amount  of  effort  and  resources  that  go  into  collecting  vendor  reports  have  become 
inordinate  in  many  research  libraries.  This  is  an  unintended  consequence  as  libraries'  collections 
are  becoming  distributed  over  the  network.  Even  a well-intentioned  effort  to  share  databases  can 
result  in  incomplete  and  inconsistent  data  that  are  difficult  to  summarize  and  use.  As  we  have 
shown  in  the  vendor  report  comparison  section,  there  is  a wide  variation  not  only  in  the  kinds  of 
measures  being  reported  but  also  the  way  reports  are  formatted  and  delivered  to  libraries. 

Before  we  proceed  with  meeting  with  vendors,  it  is  absolutely  critical  that  the  research  library 
community  work  together  and  agree  on  the  core  set  of  measures  required  from  database  vendors 
and  the  data  delivery  methods.  This  work  will  build  upon  initiatives  like  the  ICOLC  Guidelines 
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for  Statistical  Measures  of  Usage  of  Web-based  Indexed,  Abstracted,  and  Full  text  Resources , as 
well  as  other  projects  in  process. 

The  ARL  Working  Group  on  Database  Vendor  Statistics  (WG)  under  the  E-Metrics  Project 
will  be  used  as  a primary  channel  and  catalyst  for  the  effort.  During  the  next  Phase  of  the  study 
the  WG  will  develop  a written  statement  discussing  their  needs  for  statistics  and  measures, 
propose  some  standardized  definitions  for  selected  statistics  and  measures,  and  begin  discussions 
with  selected  vendors  on  these  needs. 

Continued  Sharing  of  Project  Information 

There  is  a significant  overlap  in  terms  of  what  each  library  is  doing  to  collect  and  process 
various  networked  statistics  and  measures.  For  example,  with  regard  to  vendor  reports,  research 
libraries  all  contact  more  or  less  the  same  vendors  for  usage  reports,  receive  the  reports  in  the 
same  format,  process  them  in  hopes  to  summarize  and  extract  intelligence  from  the  data.  There  is 
a significant  overlap  of  effort  to  facilitate  and  automate  the  process.  On  the  other  hand,  there  are 
also  considerable  differences  among  participants  as  to  their  data  collection  and  reporting 
activities.  The  study  team  will  continue  to  post  such  information  on  the  list  (arl- 
emetrics@arl.org)  and  look  forward  to  feedback  from  participating  libraries  regarding  the  Phase  I 
report. 


Develop  a Set  of  Core  Performance  Measures  for 
Networked  Resources  and  Services 

A written  methodology  will  be  completed  to  propose  and  develop  these  measures  and  field 
test  them  at  selected  participant  libraries.  Separately  a short  written  manual  will  be  developed 
and  describe  the  data  collection  and  analysis  techniques  and  methods  for  each  performance 
measure.  Currently,  the  study  team  anticipates  that  the  initial  core  statistics  and  measures  will 
result  from  findings  in  the  Phase  I report.  We  also  anticipate  ongoing  discussions  with  lead 
participants  as  to  the  development  and  refinement  of  these  statistics  and  measures.  Finally,  it 
should  be  pointed  out  that  the  development  of  these  statistics  and  measures  will  need  to  be 
coordinated  with  the  work  being  done  by  the  vendor  statistics  working  group.  An  issue  here  is 
the  degree  to  which  the  vendor  statistics  working  group  can  reach  agreement  as  to  key  measures 
and  issues  and  work  to  resolve  them  with  the  vendors. 


Meetings 

Members  of  the  study  team  and  the  project  principals  will  present  findings  from  Phase  I at  the 
CNI  Winter  meeting  in  San  Antonio,  December  7-8,  2000.  We  hope  to  obtain  feedback  and 
suggestions  from  the  project  briefing  session  to  inform  the  development  of  statistics  and 
measures.  The  project  briefing  session  will  be  organized  to  encourage  such  feedback. 

The  vendor  statistics  working  group  will  be  meeting  in  conjunction  with  the  ALA  midwinter 
meeting,  Friday,  January  12,  2001  1-4:00  PM  in  Washington,  DC.  At  this  meeting  we  hope  to 
continue  identifying  measures  and  statistics  that  the  ARL  community  believes  most  important  to 
be  produced  from  vendor-based  data.  We  also  intend  to  continue  coordinating  the  activities 
among  interested  organizations  in  developing  statistics  and  measures  for  networked  services  at 
this  meeting  and  others  being  held  in  conjunction  at  ALA  Midwinter. 
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Currently,  there  are  plans  being  developed  for  a workshop  to  be  held  in  February  2001  and 
sponsored  by  NISO  and  NCLIS  to  review  standards  for  a number  of  library  statistics  including 
network  and  vendor  statistics.  The  study  team  anticipates  that  representatives  from  ARL,  the 
study  team,  and  the  WG  will  participate  in  this  meeting. 

The  study  team  and  ARL  principals  have  scheduled  Monday,  April  9,  2001,  9:00  AM  to 
Noon  (in  conjunction  with  the  CNI  conference  in  Washington,  DC)  to  meet  with  participating  E- 
Metrics  libraries.  The  purpose  of  the  meeting  will  be  to  review  project  status,  discuss  proposed 
statistics  and  performance  measures,  and  continue  activities  on  working  with  database  vendors 
for  selected  statistics.  Specifics  about  this  workshop  and  its  content/organization  will  be 
forthcoming  as  the  project  continues. 


4.  Developing  Statistics  and  Performance  Measures  for  Networked  Services 

The  Phase  I report  suggests  that  there  is  some  considerable  activity  among  the  participating 
libraries  regarding  the  development  of  statistics  and  performance  measures  to  describe  activities 
in  the  networked  environment.  The  report  also  suggests  that  it  may  be  too  early  to  identify  “best 
practices”  until  there  is  more  knowledge  about  this  topic.  The  report  also  suggests  that  there  is 
much  work  yet  to  be  done  - and  there  are  a number  of  issues  that  will  require  additional  attention 

The  information  obtained  from  the  Phase  I effort  will  provide  an  excellent  base  to  continue 
project  activities  in  Phase  II.  As  the  project  moves  forward  it  will  be  important  for  participating 
libraries  to  take  an  active  involvement  in  the  vendor  statistics  working  group,  to  review  and 
participate  in  the  development  of  proposed  statistics  and  performance  measures,  and  to  provide 
feedback  and  suggestions  to  the  study  team  as  Phase  II  continues. 
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Appendix  A 


Usage  Measures  for  Electronic  Resources 

ARL  Project  Planning  Session. 

Regal  McCormick  Ranch  Resort 
Scottsdale,  Arizona,  February  27-29,  2000 


Introduction 


The  objective  of  Usage  Measures  for  Electronic  Recourses  ARL  Project  Planning  Session,  held  in 
Scottsdale,  Arizona,  was  to  establish  a project  on  usage  measures  for  electronic  information  resources  that 
would  help  ARL  libraries  identify  a set  of  measures  for  assessing  electronic  services  and  resources  in  an 
academic  setting. 

The  purpose  of  the  session  was  to  introduce  the  project  and  allow  libraries  to  describe  the  character 
and  nature  of  electronic  services  provided. 

Martha  Kyrillidou  and  Julia  Blixrud  prepared  and  distributed  a short  survey  before  the  project 
planning  meeting  and  compiled  the  responses.  Wonsik  "Jeff’  Shim,  project  consultant  from  Florida  State 
University,  analyzed  responses  of  the  participants  to  the  questions  and  presented  a summary  of  findings  at 
the  meeting. 


Of  35  ARL  institutions  (not  including  the  California  Digital  Library  at  the  time)  that  received  the 
survey,  22  libraries  responded  to  the  survey.  Also,  24  institutions  have  decided  to  participate  in  the 
project.  Among  the  22  responses  to  the  survey,  15  libraries  are  now  participants  in  the  project. 


The  questions  were  the  following: 


1.  What  do  you  need  to  know  to  make  good  resource  decisions  related  to  the  provision  of 
electronic  services  or  purchase  of  networked  resources?  What  information  do  you  need  to 
communicate  to  decision  makers  about  electronic  resources? 


2.  What  data  do  you  need  to  measure  the  impact  of  electronic  services  and  resources?  Do  you 
have  this  data  available  now? 


3.  What  are  you  currently  doing  at  your  institution  in  terms  of  measuring  electronic  services  and 
resources?  Describe  your  current  role  in  developing  measures  for  assessing  electronic  resources 
and  services. 

4.  What  are  your  primary  goals  in  participating  in  this  project  planning  session? 
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Findings 

The  following  summarizes  the  responses  of  the  participants  to  the  questions  that  provide  some 
indication  of  data  collection  activities  at  the  responding  libraries.  Responses  were  summarized  and  tabular 
presentations  of  findings  are  provided  following  each  question. 

1.  What  do  you  need  to  know  to  make  good  resource  decisions  related  to  the  provision  of 
electronic  services  or  purchase  of  networked  resources?  What  information  do  you  need  to 
communicate  to  decision-makers  about  electronic  resources? 

Table  1 A shows  that  there  are  several  important  issues  that  need  to  be  considered  before  making  any 
decision  related  to  provision  of  electronic  services  or  purchases  of  networked  services.  To  illustrate, 
sixteen  participants  would  like  to  see  some  kind  of  statistics  and  performance  measures  in  ARL  member 
libraries  so  that  they  can  show  "costs  and  values  of  the  services  for  their  user  community."  Also,  the  need 
for  quantitative  and/or  qualitative  data  on  users  and  their  information  seeking  behaviors  were  also  pointed 
out  by  sixteen  ARL  member  libraries'  representatives.  Fourteen  respondents  noted  that  the  nature  of  use 
and  access  was  an  important  issue  for  library  decision  makers  "to  determine  which  factors  contribute  to 
the  success  of  the  services  and  resources"  while  thirteen  individuals  paid  particular  attention  to  their  users' 
satisfaction.  Seven  people  focused  on  lack  of  uniformity  and  standardization  of  performance  measures 
because,  as  one  noted,  "the  lack  of  uniformity  or  consistency  among  electronic  resource  providers  makes 
their  usage  data  less  valuable  than  it  might  be."  As  a temporary  solution,  retrieving  the  statistics  available 
from  vendors  to  the  ARL  member  libraries  was  offered  by  ten  of  the  participants.  Finally,  in  relation  to 
the  provision  of  electronic  services  or  purchase  of  networked  resources,  five  of  the  libraries  would  like  to 
know  the  effects  of  electronic  resources  on  existing  services,  such  as  personnel  and  reference  services. 


Table  1A: 

What  do  you  need  to  know  to  make  a good  resource  decisions  related  to  the  provision  of  electronic 

services  or  purchase  of  networked  resources? 

Statistics  and  performance  measures 

16 

Users  and  their  information  seeking  behaviors 

16 

Nature  of  use  and  access 

14 

User  satisfaction 

13 

Access  to  statistics  available  from  vendors 

10 

Uniformity/standardization  of  performance  measures 

7 

Effects  of  electronic  resources  on  services  (e.g.,  personnel) 

5 

In  reference  to  the  second  part  of  the  question  as  shown  in  Table  IB,  cost  and  effectiveness  analysis 
(14)  was  the  top  issue  among  the  respondents.  Networking  and  hardware/software  infrastructure  and 
interfaces  and  ease  of  use  were  other  important  issues  highlighted  by  eleven  respondents.  Among  them, 
ten  library  representatives  would  like  to  know  whether  electronic  resources  came  with  links  with 
alternative  formats,  such  as  paper  versions.  Five  individuals  believed  that  trial  versions  and 
demonstrations  before  purchasing  and  workshops  after  purchasing  needed  to  be  provided  by  vendors. 
Concerns  over  reliability  of  vendor  services  and  sources  of  content  of  available  resources  were  pointed 
out  by  four  respondents.  Finally,  there  were  also  other  issues,  identified  by  three  representatives,  that 
ARL  members  needed  to  consider  while  making  a good  decision  about  electronic  resources  and  services: 
Accurate  product  information,  networking  and  installation  requirements,  scope  of  resources,  maintenance 
and  support  services  and  licensing  issues. 
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Table  IB: 

What  do  we  need  to  communicate  to  decision-makers  about  electronic  resources? 

Cost  and  effectiveness  analysis 

14 

Networking  and  hardware/software  infrastructure 

11 

Interfaces  and  ease  of  use 

11 

Links  with  alternative  formats  (such  as,  paper  version) 

10 

Workshops,  trials  and  demonstrations 

5 

Reliability  of  vendor  services 

4 

Sources  of  content 

4 

Accurate  product  information 

3 

Scope  of  electronic  services 

3 

Maintenance  and  support  services 

3 

Licensing  issues 

3 

2.  What  data  do  you  need  to  measure  the  impact  of  electronic  services  and  resources?  Do  you  have 
this  data  available  now? 


Table  2A: 

What  data  do  you  need  to  measure  the  impact  of  electronic  services  and  resources? 

Number  of  use 

14 

Number  of  users 

13 

Number  of  logins  and  denials 

12 

Number  of  hits  (Full-text  vs.  index) 

9 

Number  of  downloads 

8 

Number  of  searches 

6 

Response  and  access  time 

6 

Number  of  reports 

3 

Service  costs 

3 

Number  of  prints 

2 

Number  of  reference  services 

2 

Staff 

2 

Table  2A  lists  the  issues  related  to  measuring  the  impact  of  electronic  services  and  resources  among 
the  ARL  member  libraries.  Fourteen  of  the  members  would  like  to  know  the  number  of  uses  through 
electronic  services  and  resources.  Also,  number  of  users  (13)  and  number  of  logins  and  denials  (12)  were 
two  topics  on  which  participants  heavily  agreed.  Particularly,  the  library  representatives  were  interested 
in  knowing  the  number  of  hits  (9),  number  of  downloads  (8),  number  of  searches  (6),  and  response  and 
access  times  (6).  Three  respondents  needed  to  know  periodically  reporting  option  of  the  resources  and 
service  costs  of  electronic  services.  Finally,  two  respondents  wanted  to  know  number  of  prints,  number 
of  reference  services  and  the  impact  on  staff  for  electronic  services  and  resources  in  ARL  member 
libraries. 

Among  the  respondents,  eight  of  the  members  have  some  data  about  electronic  services  and  resources 
although  fourteen  of  them  had  no  data  available  on  the  services  and  resources  provided  (see  Table  2B). 
Those  who  have  the  data  complained  that  the  data  are  not  consistent  across  providers  or  products.  They 
seem  to  agree  on  that  "the  usefulness  of  data  available  is  hampered  by  lack  of  consistency  across 
resources,  lack  of  comparable  data  for  print  resources  and  offline  services,  lack  of  consistent  longitudinal 
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data/studies,  lack  of  consistency  across  institutions,  and  lack  of  recognized  methodologies  for  evaluation." 
In  addition,  some  suggested  that  the  ICOLC  guidelines  for  statistical  measures  of  usage  of  web-based 
indexed,  abstracted,  and  full  text  resources  could  be  a workable  guide  until  industry  standards  are 
developed  to  measure  performance  and  outcomes  for  electronic  resources. 


Table  2B:  Do  you  have  this  data  available  now? 

Respondents  that  have  data 

8 

Respondents  that  do  not  have  any  data 

14 

3.  What  are  you  currently  doing  at  your  institution  in  terms  of  measuring  electronic  services  and 
resources?  Describe  your  current  role  in  developing  measures  for  assessing  electronic  resources 
and  services. 

The  participants  have  been  measuring  and  looking  for  some  indication  of  uses,  costs,  titles,  logins  and 
tumaways,  and  receiving  feedback  from  staff  and  users  for  electronic  services  and  resources  (see  Table 
3 A).  Specifically,  thirteen  were  counting  uses  and  accesses  in  their  libraries  although  ten  of  them  were 
gathering  the  data  from  vendors,  as  they  needed.  Cost  and  benefit  analysis  was  another  activity  among 
five  of  respondents.  Also,  five  were  dealing  with  counting  titles  and  number  of  logons  and  tumaways. 
Finally,  three  respondents  were  surveying  their  users  and  looking  for  feedback  from  their  staff  as  a way  of 
measuring  their  electronic  services  and  resources  available. 


Table  3A:  What  are  you  currently  doing 

at  your  institution  in  terms  of  measuring  electronic  services  and  resources? 

Counting  uses  and  access 

13 

Gathering  data  from  vendors 

10 

Cost  and  benefit  analysis 

5 

Counting  titles 

5 

Number  of  logons  and  tumaways 

5 

Surveying  users  and  receiving  feedback  from  staff 

3 

Table  3B:  Describe  your  current  role 
in  developing  measures  for  assessing  electronic  resources  and  services 

Evaluation  analyst  role 

8 

Budgeting  role 

3 

Benchmarking  role 

2 

Acquisitions  librarian  or  collection  manager  role 

2 

Coordinator  role 

2 

Other  roles 

5 

As  shown  in  Table  3B,  while  measuring  their  electronic  services  and  resources,  the  respondents 
described  themselves  differently.  For  instance,  eight  of  the  respondents  described  themselves  as 
evaluation  analysts  while  three  stated  that  they  carried  budgeting  role  in  their  settings.  Benchmarking, 
acquisitions  librarians  or  collection  manager,  or  coordinator  roles  for  measuring  these  resources  and 
services  were  defined  by  two  participants.  Finally,  five  responses  described  themselves  in  other  roles, 
such  as  web  manager  and  other  managerial  roles. 


4.  What  are  your  primary  goals  in  participating  in  this  project  planning  session? 
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This  question  was  aimed  at  finding  out  the  reasons  of  participation  in  the  ARL  Project 
Planning  Session  and  identifying  the  primary  goals  for  participation.  According  to  the  responses  to  the 
question  (see  Table  4),  fourteen  of  the  participants  were  interested  in  participation  in  development  of 
industry  standards  to  measure  performance  and  outcomes  for  electronic  resources.  Five  would  like  to 
benchmark  their  activities  to  other  ARL  member  libraries  if  ARL  members'  shared  their  experience  with 
others  as  stated  by  four  of  the  respondents.  Finally,  two  participants  were  interested  in  library 
effectiveness  and  quality  as  a goal  in  participation  in  the  session.  In  conclusion,  as  the  findings  indicated, 
the  members  would  very  much  liked  to  see  a set  of  measures  for  assessing  electronic  services  and 
resources  in  an  academic  setting. 


Table  4: 

What  are  your  primary  goals  in  participating  in  this  project  planning  session? 

to  participate  in  development  of  industry  standards  to 
measure  performance  and  outcomes  for  electronic 
resources 

15 

to  benchmark 

5 

to  share  other  ARL  members’  experience 

4 

interested  in  library  effectiveness  and  quality 

2 
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Appendix  B 

E-Metrics  Project 

ARL  Survey  on  Statistics  and  Measures 
for  Networked  Services  and  Resources 


The  purpose  of  this  survey  is  to  gather  specific  information  about  current  data  collection  and  use  practices  in  the 
area  of  electronic  resources  and  services  among  the  participants  of  ARL's  E-metrics  project  to  inform  the 
investigators  as  they  prepare  various  site  visits.  This  preliminary  survey  will  help  the  study  team  identify  the 
existing  data  gathering  effort  that  need  to  be  studied  further  and  documented  in  the  best  practices  report.  Follow 
up  site  visits  are  expected  to  be  conducted  to  select  institutions  in  the  month  of  August. 

The  survey  is  being  conducted  by  Chuck  McClure<cmcclure@lis.fsu.edu>  and  Wonsik  "Jeff'  Shim 
<wshim@lis.fsu.edu>  at  Information  Use  Management  and  Policy  Institute  at  Florida  State  University.  If  you 
have  any  questions  about  this  survey,  please  send  us  an  email  at  wshim@lis.fsu.edu. 

The  survey  will  be  best  responded  to  by  a librarian  or  a staff  member  who  is  knowledgeable  about  statistics  and 
measures  relating  to  electronic  services  and  who  is  involved  in  the  process  of  gathering  and  processing  that 
information.  The  survey  is  not  intended  to  gather  an  exhaustive  list  of  measures  of  electronic  resources  and 
services  but  rather  representative  statistics  and  measures  your  library  is  collecting. 

You  are  receiving  this  survey  because  your  institution  has  agreed  to  participate  in  the  E-Metrics  project.  By 
answering  survey  questions,  you  are  agreeing  to  participate  in  the  project.  Collected  information  will  appear  in 
published  results  only  in  aggregate  statistics  or  form.  The  identities  of  the  respondents  will  be  used  only  for  the 
purpose  of  establishing  contacts  with  the  libraries  for  follow-up  phone  interviews  and/or  site  visits.  The 
information  will  be  kept  in  a secure  place  and  destroyed  no  later  than  December  31,  2006,  five  years  after 
completion  of  the  study. 

Your  participation  in  this  survey  is  very  important.  Thank  you  for  your  support  for  this  project. 


Please  return  the  survey  by  August  1 1,  2000  by  filling  in  your  responses  in  the  attached  Microsoft  Word 
document  or  RTF  (rich  text  format)  file.  Please  send  the  file  as  an  email  attachment  to  wshim@lis.fsu.edu. 
Finished  survey  can  also  be  faxed  to  (850)644-6253.  Attn  Wonsik  "Jeff'  Shim 
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Part  1 . Data  Collection 

Before  you  begin,  please  tell  us  about  yourself  so  we  may  contact  you  in  the  future. 
Institution: 

Name: 

Position/Title: 

Email  Address: 

Phone  Number: 


Please  indicate  the  types  of  data  elements  or  statistics  your  library  gathers,  along  with  the  source  and  the  frequency 
of  data  collection.  When  you  list  the  measures,  please  use  the  terminology  as  you  use  it  in  your  library.  If  you  need 
additional  space,  use  the  back  of  the  page  or  insert  additional  rows  in  the  following  Word  Table. 


Collected  Data 

Frequency  of  data 
collection 

Began 

Collecting  Since 

Is  this  required? 

Patron  Accessible  Resources  : E.g.  # of 
electronic  database  titles  served,  # of  library 
web  pages  in  service,  # of  e-books,  # of  full- 
text  e-joumals,  # of  librarians  providing 
electronic  reference. 

Weekly,  Monthly, 
Quarterly,  Yearly 
(Check) 

W M Q Y 

(approximately) 

month/year 

Is  the  measure  requested  by 
anyone?  For  instance, 
university  administration  or 
outside  funding  source. 
No  If  Yes,  by  whom? 

Patron  Use  : E.g.  # of  logins  or  visits,  # of 
library  web  pages  accessed,  # of  documents 
downloaded,  # of  electronic  reference 
service  transactions. 

Frequency 

(Check) 

W M Q Y 

(approximately) 

month/year 

No  If  Yes,  by  whom? 
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Collected  Data 

Frequency  of  data 
collection 

Began 

Collecting  Since 

Is  this  required? 

Users  : E.g.  % of  undergrads  who  have  used 
the  e-books,  % of  grads  who  have  used  the 
electronic  reference  service  (#  of  users  in  a 
group  by  type  of  services) 

Frequency 

(Check) 

W M Q Y 

(approximately) 

month/year 

No  If  Yes,  by  whom? 

Cost : E.g.  $ per  electronic  document 
delivered,  cost  of  database  subscription  fee 
(average  or  total),  expenditures  for 
electronic  journals  (ongoing,  leases, 
subscriptions), 

Frequency 

(Check) 

W M Q Y 

(approximately) 

month/year 

No  If  Yes,  by  whom? 

Other  : Beyond  the  above  measures,  please 
indicate  below  if  your  library  collects  other 
types  of  data  such  as  service  quality,  the 
effects  of  library  use  on  research  and 
instruction  or  the  percent  of  library  users 
satisfied  with  libraries  services. 

Frequency 

(Check) 

W M Q Y 

(approximately) 

month/year 

No  If  Yes,  by  whom? 

Part  2.  Data  Use 


1 . With  whom  do  you  share  the  collected  data  on  electronic  resources  and  services?(Check  all  that  apply) 
[ ] Among  Library  Managers  [ ] Among  Library  Staff 

[ ] With  University  Administrators  [ ] Others  (Specify) 
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2.  Please  indicate  the  kinds  of  decision  making  for  which  collected  data  is  being  used.  (E.g.  journal 
renewal/cancellation,  redesign  of  library  webpages) 


Part  3.  Issues 


Please  list  the  three  most  important  issues  your  library  faces  in  the  collection  of  statistics  and  measures  to  describe 
networked  services  and  resources. 

1. 


2, 


3, 


Part  4.  Participation  in  the  Benchmarking  Process 

Please  indicate  below  if  your  library  wants  to  be  considered  as  a best  practice  in  one  or  more  areas  in  which  the 
library  has  been  collecting  data  and  has  a systematic  process  put  in  place  for  some  time  (6+  months). 


Users  of  networked  information  services  and  resources  [ 

Uses  of  networked  information  services  and  resources  [ 

Quality  of  networked  information  services  and  resources  [ 

Cost  of  networked  information  services  and  resources  [ 

Staffing  and  training  [ 

[ 

[ 


A member  of  the  study  team  may  contact  you  for  a follow  up  interview  and/or  site  visit. 

Part  5.  Sample  reports 


Please  attach  any  sample  reports  and  data  compilations  of  statistics  and  measures  for  describing  electronic  services 
and  resources.  You  can  also  fax  them  to  Wonsik  "Jeff'  Shim  at  (850)644-6253. 


THANK  YOU  FOR  YOUR  SUPPORT. 
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Appendix  C 

(available  at  http://www.librarv.vale.edu/consortia/webstats.html) 


International  Coalition  of  Library  Consortia  (ICOLC) 

GUIDELINES  FOR  STATISTICAL  MEASURES  OF  USAGE  OF  WEB-BASED 
INDEXED,  ABSTRACTED,  and  FULL  TEXT  RESOURCES 

(November  1998) 


INTRODUCTION 

The  use  of  licensed  electronic  information  resources  will  continue  to  expand  and  in  some  cases 
become  the  sole  or  dominant  means  of  access  to  content.  The  electronic  environment,  as 
manifested  by  the  World  Wide  Web,  pro  vides  an  opportunity  to  improve  the  measurement  of  the 
use  of  these  resources.  In  the  electronic  arena  we  can  more  accurately  determine  which 
information  is  being  accessed  and  used.  Without  violating  any  issues  of  privacy  or 
confidentiality  we  can  dramat  ically  enhance  our  understanding  of  information  use. 

The  participating  consortia  of  the  ICOLC  have  a responsibility  to  their  library  members  to  ensure 
the  provision  of  usage  information  of  licensed  electronic  resources.  Information  providers  should 
want  the  same  information  to  better  understand  the  market  f or  their  services  as  well  as  to  create 
an  informed  customer  base.  These  mutual  interests  can  be  best  met  by  defining  and  creating  a 
common  set  of  basic  use  information  requirements  that  are  an  integral  and  necessary  part  of  any 
electronic  product  offering  . These  requirements  apply  to  vendor  operated  web  sites  and  to 
software  provided  to  libraries  or  consortia  for  local  operation.  Information  providers  are 
encouraged  to  go  beyond  these  minimal  requirements  as  appropriate  for  their  specific  electronic 
resources. 

These  ICOLC  guidelines  draw  heavily  upon  the  guidelines  developed  by  the  JSTOR  Web 
Statistics  Task  Force:  David  Farrell,  Berkeley,  Chair;  Jim  Mullins,  Villanova;  Kimberly  Parker, 
Yale;  Dave  Perkins,  CSU-Northridge;  Sue  Phillips,  Texas;  Camille  Wanat,  Berkeley;  Kristen 
Garlock,  JSTOR,  ex-officio.  The  ICOLC  guidelines  reflect  modifications  to  maximize  their 
broad  applicability  to  the  diversity  of  resources  licensed  by  many  ICOLC  members. 


1.  REQUIREMENTS 

Each  use  element  defined  below  should  be  able  to  be  delineated  by  the  following  subdivisions; 

1 . By  each  specific  database  of  the  provider 

2.  By  each  institutionally-defined  set  of  IP  addresses  / locators  to  subnet 
level 


3.  By  total  consortium 
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4.  By  special  data  element  passed  by  subscribe^  e.g.,  account  or  ID  number) 

5.  By  time  period.  Vendor’s  system  should  minimally  report  by  month.  For 
each  month,  each  type  of  use  should  be  reported  by  hour  of  the  day,  and 
vendor  should  maintain  24  months  of  historical  data 

Use  Elements  that  must  be  provided  are: 

Number  of  queries  (Searches)  categorized  as  appropriate  for  the  vendor’s 
information.  A search  is  intended  to  represent  a unique  intellectual 
inquiry.  Typically  a search  is  recorded  each  time  a search  form  is 
sent/submitted  to  t he  server.  Subsequent  activities  to  review  or  browse 
among  the  records  retrieved  or  the  process  of  isolating  the  correct  single 
item  desired  do  not  represent  additional  searches,  unless  the  parameter(s) 
defining  the  retrieval  set  is  modified  through  resubmission  of  the  search 
form,  a combination  of  previous  search  sets,  or  some  other  similar 
technique. 

Number  of  Menu  Selections  categorized  as  appropriate  to  the  vendor’s 
system.  If  display  of  data  is  accomplished  by  browsing  (use  of  menus), 
this  measure  must  be  provided  (e.g.  an  electronic  journal  site  provides 
alphabetic  and  subject-based  menu  options  in  addition  to  a search  form. 
The  number  of  searches  and  the  number  of  alphabetic  and  subject  menu 
selections  should  be  tracked). 

Number  of  sessions  (Logins),  if  relevant,  must  be  provided  as  a measure 
of  simultaneous  use.  It  is  not  a substitute  for  either  query  or  menu 
selection  counts. 

Number  of  tum-aways,  if  relevant,  as  a contract  limit  (e.g.,  requests 
exceed  simultaneous  user  limit). 

Number  of  items  examined  (i.e.,  viewed,  marked  or  selected,  downloaded, 
emailed,  printed)  to  the  extent  these  can  be  recorded  and  controlled  by  the 
server  rather  than  the  browser: 

1 . Citations  displayed  (for  A&I  databases) 

2.  Full  text  displayed  broken  down  by  title,  ISSN  with  title  listed,  or  other 
title  identifier  as  appropriate 

1 . Tables  of  Contents  displayed 

2.  Abstracts  displayed 

3.  Articles  or  essays,  poems,  chapters,  etc.,  as  appropriate, 
viewed  (e.g.,  ASCII  or  HTML)  or  downloaded  (e.g.  PDF, 
email) 
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4.  Other  (e.g.,  image  / AV  files,  ads,  reviews,  etc.,  as 
appropriate) 

The  ICOLC  is  preparing  a separate  guideline  on  Technical  Performance  of  Web-based  Services 
for  reporting  of  system  related  parameters  ( e.g.,  downtime,  response  time). 

2.  PRIVACY  AND  USER  CONFIDENTIALITY:  Statistical  reports  or  data  that  reveal 
confidential  information  about  individual  users  must  not  be  released  or  sold  by  information 
providers  without  permission  of  the  consortium  and  its  member  libraries. 

3.  INSTITUTIONAL  OR  CONSORTIAL  CONFIDENTIALITY:  Providers  do  not  have  the  right 
to  release  or  sell  statistical  usage  information  about  specific  institutions  or  the  consortium 
without  permission,  except  to  the  consortium  administrators  and  member  libraries.  Use  of 
institutional  or  consortium  data  as  part  of  an  aggregate  grouping  of  similar  institutions  for 
purposes  of  comparison  does  not  require  prior  permission  as  long  as  specific  institutions  or 
consortia  are  not  identifiable.  When  required  by  contractual  agreements,  information  providers 
may  furnish  institutional  use  data  to  the  content  publishers. 

4.  COMPARATIVE  STATISTICS:  Information  providers  should  provide  comparative  statistics 
that  give  consortia  a context  in  which  to  analyze  statistics  at  the  aggregate  institutional 
(consortium  member)  level.  For  example,  a grouping  for  purposes  of  comparison  should  be 
compiled  by  the  information  provider  (e.g.,  statistics  from  an  anonymous  selection  of  similar 
institutions),  or  it  might  be  a grouping  composed  on  demand  (e.g.,  statistics  from  all  campuses  in 
a consortium,  presented  either  anonymously  or  not,  as  desired  by  the  participating  institutions). 

5.  ACCESS  / DELIVERY  MECHANISMS  / REPORT  FORMATS:  Access  to  statistical  reports 
should  be  provided  via  web-based  reporting  systems  and  be  restricted  by  IP  address  or  another 
form  of  security  such  as  passwords.  Institutions  should  be  able  to  authorize  access  to  their  use 
data  by  other  institutions  in  the  consortium  if  they  desire. 

Information  providers  should  maintain  access  to  tabular  statistical  data  through  their  web  site 
(updated  monthly)  which  a participant  can  access,  aggregate  and  manipulate  on  demand.  When 
appropriate,  these  data  also  should  be  available  in  flat  files  containing  specified  data  elements 
that  can  be  downloaded  and  manipulated  locally.  Information  providers  are  also  encouraged  to 
present  data  as  graphs  and  charts. 

Adopters  of  This  Statement 

This  statement  was  adopted  in  principle  by  member  representatives  of  the  "International 
Coalition  of  Library  Consortia"  (ICOLC)  whose  institutions  are  listed  below.  This  statement 
does  not  necessarily  represent  the  official  views  of  each  consortium  listed.  All  consortia  listed 
are  in  the  United  States  unless  otherwise  noted. 

Consortia  whose  member  representatives  adopted  this  statement: 

ALICE  (the  Appalachian  Library  Information  Cooperative) 

AMIGOS  Bibliographic  Council,  Inc. 

Arizona  Universities  Library  Consortium  (AULC) 

BCR 

Big  12  Plus 

Boston  Library  Consortium 

British  Columbia  Electronic  Library  Network  (Canada) 

California  Digital  Library  (CDL) 

The  California  State  University. 

Canadian  National  Site  Licensing  Project  (CNSLP) 

Colorado  Alliance 
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Committee  for  Institutional  Cooperation  (CIC) 

CAUL  (Council  of  Australian  University  Librarians) 

Council  of  Prairie  and  Pacific  University  Libraries  (COPPUL)  (Canada) 

CURL  (United  Kingdom) 

Florida 

Center  for  Library  Automation 
GALILEO 

Illinois  Cooperative  Collection  Management  Program  (ICCMP) 

Illinois  Library  Computer  Systems  Organization  (ILCSO) 

INCOLSA 

Israel  Center  for  Digital  Information  Services 

Louisiana  Library  Network 

MERLIN 

Michigan  Library  Consortium 

MINITEX  Library  Information  Network 

MIRACL 

MOREnet 

NERL 

Netherlands  Association  of  University  Libraries,  Royal  Library,  and  Library  of  the  Royal  Academy  of  Sciences 
(Dutch  acronym:  UKB) 

Network  of  Alabama  Academic  Libraries 

New  England  Law  Library  Consortium  (NELLCO) 

Novanet  (Canada) 

OhioLINK 

Ontario  Council  of  University  Libraries  (OCUL) 

Orbis 

PAUNET 

Pennsylvania  Academic  Library  Consortium,  Inc.  (PALCI) 

Pioneer,  Utah’s  Online  Library 
PORTALS 

SCELC  - Southern  California  Electronic  Library  Consortium 
SCONUL  (Ireland  and  United  Kingdom) 

Solinet 

Southeastern  Wisconsin  Information  Technology  Exchange  (SWITCH) 

Sub-Committee  on  Libraries  of  the  Conference  of  Rectors  and  Principals  of  Universities  of  Quebec  (CREPUQ). 
TexShare 

TriUniversity  Group  of  Libraries  (TUG)  (Canada) 

The  Triangle  Research  Libraries  Network  (TRLN) 

UNILINC  (Australia) 

University  of  Texas  System  Digital  Library 
Utah  Academic  Library  Consortium 
VIVA  (The  Virtual  Library  of  Virginia) 

Washington  Research  Library  Consortium 
Wisconsin  InterLibrary  Services  (WILS) 

About  the  International  Coalition  of  Library  Consortia  (ICOLC) 

The  International  Coalition  of  Library  Consortia  (ICOLC)  first  met  as  the  "Consortium  of 
Consortia"  (COC)  in  1996.  The  Coalition  is  an  international,  informal  group  currently 
comprising  over  ninety  library  consortia  in  North  America,  Europe  , Australia,  Israel,  China,  and 
South  Africa.  The  coalition  membership  serve  primarily  higher  education  institutions  by 
facilitating  discussion  among  consortia  on  issues  of  common  interest.  The  ICOLC  conducts 
meetings  throughout  the  year  dedicated  to  kee  ping  its  members  informed  about  new  electronic 
information  resources,  pricing  practices  of  electronic  providers  and  vendors,  and  other  issues  of 
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importance  to  consortia  directors  and  governing  boards.  The  Coalition  also  meets  with  the 
information  provide  r community,  creating  a forum  for  discussion  about  product  offerings  and 
issues  of  mutual  concern. 

More  information  about  ICOLC  can  be  found  at  http://www.library.yale.edu/consortia. 

For  further  information  about  this  document,  please  contact: 

• Tom  Sanville,  Executive  Director , Ohio  LINK 

Suite  300,  2455  North  Star  Road,  Columbus,  OH  43221 
Phone:  614-728-3600,  ext.  322 
Email:  tom@ohiolink.edu 
Fax:614-728-3610 

• Sue  Phillips,  Director,  University  of  Texas  System  Digital  Library,  University  of  Texas 

P.  O.  Box  P,  Austin,  TX  78713-8916 
Phone:  512-495-4350 
Email:  s.phillips@mail.utexas.edu 
Fax:  512-495-4347 
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Appendix  D 

(available  at  http://www.librarv.vale.edu/consortia/webstats.html) 

Electronic  Vendor  Statistical  Reporting  Capabilities 

Describe  the  statistics  you  provide,  and  discuss  whether  your  statistical  reporting  complies  with  guidelines 
developed  by  the  International  Coalition  of  Library  Consortia,  which  may  be  found  at 
www.librarv.vale.edu/consortia/webstats.html. 


Ancestry.com  (Spring  2000) 

We  are  currently  working  on  statistics  program  that  will  allow  your 
library  to  see  stats  on  a daily,  weekly,  monthly,  and  annual  basis.  At  the 
moment,  we  are  looking  at  a company  called  SuperStats.com 
(<http://www.superstats.com/>).  which  will  allow  us  to  provide  any 
statistical  data  needed  for  our  clients.  We  will  keep  our  subscribers 
posted  on  the  progress  of  this  option. 

Ardenonline  - Works  of  Shakespeare 
(Fall  1999) 

We  can  provide  statistics  in  accordance  with  the  ICOLC  guidelines  if 
required. 

Aries  Systems  (Spring  2000) 

For  each  institutional  customer  we  record  aggregated  statistical 
information  such  as:  date  and  length  of  search  sessions,  number  of 
databases  and  thesauri  searches,  documents  (abstracts)  and  summaries 
viewed.  These  statistics  are  available  upon  request.  However,  for 
reasons  of  privacy,  we  do  not  track  or  record  any  individual  search 
strings. 

Baker  and  Taylor  EBIS  (Spring  2000) 

Statistics  provided  are  related  to  user  id  accession  of  the  database.  The 
IP  address  method  of  using  the  database  cannot  be  traced  for  certain 
aspects  of  reporting.  Statistics  can  include,  but  are  not  limited  to  the 
following:  amount  of  searches,  types  of  searches,  last  login,  amount  of 
carts,  amount  of  titles  in  a cart,  amount  of  dollars  allocated  per  cart,  and 
amount  of  discounted  dollar  amount  allocated  per  cart.  In  reviewing  the 
International  Coalition  of  Library  Consortia,  we  comply  in  part  with  the 
guidelines  addressed  in  their  web  page. 

Bell  & Howell  (Spring  2000) 

We  provide  monthly  reports  that  allow  libraries  to  review  their 
ProQuest  activity.  These  usage  statements  provide  statistical 
information  about  the  number  of  articles  (from  each  database  to  which 
the  library  subscribes,  and  by  each  title  in  that  database)  delivered 
electronically  to  users  for  viewing  or  emailing.  You  can  represent 
individual  libraries,  or  group  them  (for  example,  by  county)  in  whatever 
fashion,  and  to  whatever  level  of  detail,  you  like. 

We  are  working  to  develop  these  reports  in  accordance  with  the  ICOLC 
guidelines.  Compliance  with  these  guidelines  is  underway,  and  planned 
for  completion  by  Summer  2000.  Currently,  ProQuest  provides  the 
following  compliant  functionality. 

• usage  reports  available  online  with  an  administrator's  account 
(similar  to  the  way  our  customers  currently  access  the  Library 
Holdings  function) 

• reports  available  on  demand  (with  information  updated 
monthly) 

• summary  reports  delivered  in  HTML 

• detailed  reports  e-mailed  in  ASCII  delimited  text 
The  following  functionality  is  planned  for  Summer  2000. 

• reports  expanded  to  include  information  on  searches 

• cumulative  reports  (for  example,  quarterly,  annually) 

• inclusion  of  full  journal  title  vs.  abbreviated  version 
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• ISSN  added 

• daily  updates  to  usage  statistics  (month-to-date) 

Big  Chalk  (Spring  2000) 

Reports  detailing  usage  by  building  and  cumulative  usage  by  group  or 
region  are  available.  Usage  Reports  can  be  supplied  monthly  or 
quarterly.  Currently,  the  state  or  regional  sales  representative  supplies 
usage  reports.  Access  to  usage  statistics  via  the  World  Wide  Web  is 
currently  under  development  dependent  on  the  username  of  the  account. 
For  security  reasons,  to  access  any  usage  report  a username  and 
password  will  be  required.  Statistic  Definitions:  The  following  are  short 
definitions  for  each  of  the  statistical  terms  used  in  a returned  usage 
report.  These  usage  terms  consist  of  total  queries,  total  retrievals,  and 
hours  connected.  The  following  definitions  are  compliant  with  the 
definitions  set  by  The  International  Coalition  of  Library  Consortia 
(ICOLC).  Queries:  A query  or  search  is  defined  as  each  time  a search 
form  is  sent  to  the  server  (each  time  the  search  button  is  clicked). 
Browsing  through  retrievals  from  a single  search  do  not  represent 
additional  searches.  If  a query  is  refined  by  altering  the  query  phrase  or 
placing  additional  search  parameters  on  the  search  form,  an  additional 
query  is  registered.  Retrievals:  A retrieval  is  recorded  each  time  the  user 
clicks  or  opens  a document  on  the  result  list.  Additional  functions  of 
printing,  saving,  or  e-mailing  the  document  are  not  recorded  as 
additional  retrievals.  Hours:  "Hours  Connected"  is  defined  as  the 
amount  of  time  an  account  is  logged  on  to  Electric  Library.  This 
number  is  recorded  in  hours  and  is  rounded  to  the  nearest  whole  hour. 
This  number  does  not  represent  individual  sessions,  but  instead  a 
cumulative  account  of  access  to  eLibrary  for  a single  username. 

Bowker  - Books  in  Print  (Spring  2000) 

Bowker  takes  the  ICOLC  "Guidelines  for  Usage  Statistics"  very 
seriously,  and  is  working  toward  fulfilling  as  many  of  those  Guidelines 
as  possible  as  the  product  develops.  Current  booksinprint.com 
statistical  report  functionality:  time  reporting  for  the  calendar  month, 
average  usage  by  day  of  the  week  (e.g.,  Mondays,  Tuesdays),  average 
usage  by  hour  of  the  day  (e.g.,  1 2:00  pm,  1 :00  pm,  2:00  pm),  broken 
down  by  individual  IP  address  and  usemame/password.  The  statistics 
will  be  compiled  on  a monthly  basis,  with  a rolling  six-month  archive. 
Bowker  is  developing  functionality  to  include  on-screen  displays  of 
logins,  title  searches,  author  searches,  ISBN  searches,  etc. 

Bowker  - Ulrich  (Spring  2000) 

Bowker  takes  the  ICOLC  "Guidelines  for  Usage  Statistics"  very 
seriously,  and  is  working  toward  fulfilling  as  many  of  those  Guidelines 
as  possible  as  the  product  develops.  The  current  version  of 
ulrichsweb.com  does  not  provide  for  usage  statistics.  However,  version 
2.0,  expected  to  be  released  this  summer,  will  provide  site  location  login 
numbers  and  percentages,  average  daily  activity  (logins  by  site  location, 
searches  by  site  location,  browses  by  site  location  and  detailed  views 
accessed  by  site  location),  and  average  hourly  activity  (average  number 
of  searches  by  hour  of  the  day). 

Britannica  (Fall  1999) 

Usage  statistics  for  Britannica  Online  for  each  institution  are  available 
to  the  Britannica  contact  at  the  subscribing  institution,  displaying  daily 
and  running  totals  of  the  numbers  of  Britannica  URLs  accessed  as 
Queries  ("Why  is  the  sky  blue?"),  Documents  (articles  opened),  and 
Other  (images,  etc.).  Usage  statistics  are  available  for  the  IP  addresses 
specified  by  the  institution  or  by  the  Usemame/Password  assigned  in  the 
Britannica  cookie-based  authentication.  A customer-accessible  Web 
system  is  available,  and  currently  being  used  by  most  of  the  Consortium 
Adopters  of  these  standards. 

Cambridge  Scientific  Abstracts  (Spring 

CSA  provides  online  usage  statistics  for  all  IDS  accounts,  which 
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2000)  Cambridge  Scientific  Abstracts 
(cont.) 

comply  with  the  ICOLC  guidelines.  Each  customer  can  access  their 
own  usage  report.  Access  to  these  usage  reports  is  restricted  by  IP 
address  and  usemame  and  password.  The  usage  report  summarizes 
monthly  usage  in  three  parts: 

1 . The  first  part  reports  the  number  of  times  your  usemame  and 
password  were  used  to  access  the  Internet  Database  Service  each  month. 
(Earliest  available  data  is  from  September  1996) 

2.  The  second  part  shows  a count  of  the  number  of  searches  that  were 
submitted  by  persons  accessing  with  your  usemame  and  password. 
(Earliest  available  data  is  from  April  1997) 

3.  The  third  part  reports  the  number  of  times  each  database  was 
searched.  Since  it  is  possible  to  search  more  than  one  database  with 
each  submitted  search,  this  count  will  be  higher  than  the  number  of 
searches  submitted.  (Earliest  available  data  is  from  September  1996). 

CARL’s  Dialog  Basic  Collection  (Spring 
2000) 

Dialog(2)CARL  usage  reports  are  available  via  e-mail  on  a weekly  or 
monthly  basis  or  both.  The  reports  include:  total  searches  (broken  down 
by  number  of  searches  with  hits,  number  of  searches  with  zero  hits,  and 
total  of  both);  total  number  of  sessions;  number  of  tumaways  due  to  all 
ports  in  use;  individual  database  usage  (number  of  searches  with  hits  in 
each  database  accessed,  number  of  searches  with  zero  hits,  and  total  of 
both  per  database);  and  number  of  searches  by  domain/IP  address. 
Statistics  can  also  be  sent  in  comma-delimited  format  if  the  library 
wishes  to  import  and  manage  the  data  in  its  own  spreadsheet  program. 
For  consortia,  the  above  statistics  in  either  standard  or  comma-delimited 
format  can  be  broken  down  by  individual  institution. 

The  Dialog@CARL  statistical  reporting  partially  complies  with  the 
guidelines  developed  by  the  International  Coalition  of  Library 
Consortia.  Dialog@CARL  provides  reporting  for  each  specific 
database,  by  each  institutionally  defined  set  of  IP  addresses,  by  total 
consortium,  and  by  special  data  element  passed  by  subscriber.  In 
addition,  number  of  queries,  number  of  sessions,  and  number  of 
tumaways  are  all  recorded. 

Chadwyck-Healy  (Fall  1999) 

We  currently  do  not  provide  usage  stats  that  are  in  line  with  the  ICOLC 
guidelines.  However,  the  in-progress  software  enhancements 
mentioned  in  item  #4  also  include  planned  statistical  reporting 
capabilities.  We  are  using  the  ICOLC  statement  as  our  guide. 

CINAHL  DIRECT  (Spring  2000) 

Cinahl  Information  Systems  provides  usage  reports  containing  the  name 
of  the  database  in  use,  the  number  of  the  concurrent  user,  the  sign  on 
and  sign  off  times,  and  the  total  elapsed  time  for  the  session.  The 
reports  can  be  generated  to  meet  your  schedule. 

EBSCO  (Spring  2000) 

EBSCOhost  provides  the  ability  to  track  usage  and  provide  statistical 
reports  on  virtually  any  client/server  communication.  The  library 
administrator  will  have  the  ability  to  access  and  control  all  user 
configurations  and  generate  reports  automatically  on-site.  Usage 
statistics  can  be  used  to  measure  how  often  titles  and  databases  are 
searched.  Title,  database  and  session  usage  statistics  can  be  collected 
for  specific  time  periods  (monthly  or  annually),  limited  to  a specific 
data  field  (i.e.  ISSN)  and/or  searched  in  a particular  way  (i.e.  a 
database  name  or  number  of  abstracts  downloaded/printed). 

1 . Title  Usage  Statistics:  Library  administrators  will  be  able  to  monitor 
exactly  which  titles  are  being  accessed  with  EBSCOhost  and  be  able  to 
determine: 

• Number  of  abstracts  browsed/printed/downloaded  per  title 

• Total  number  of  full  text  articles/pages  browsed  for  each  title 

2.  Session  Usage  Statistics:  Library  administrators  will  also  be  able  to 
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monitor  session  activity  and  be  able  to  determine:  The  number  of 
sessions  Number  of  searches  Average  number  of  hits  per  search  The 
user  profile  which  is  used  most  often 

3.  Database  Usage  Statistics:  Library  administrators  will  be  able  to 
monitor  and  track  database  usage  in  the  following  areas: 

4.  Total  number  of  searches  for  each  database 

5.  Average  number  of  hits  per  search  for  each  database 

6.  Total  number  of  abstracts  browsed/printed/downloaded  from  each 
database 

7.  Total  number  of  articles  e-mailed  for  each  database 

Electric  Library  (Spring  2000) 

Electric  Library  can  provide  reports  monthly  detailing  number  of 
queries,  number  of  retrievals,  and  time  connected  to  our  service  for  each 
institution.  Additionally,  for  each  institution,  reports  showing  the  most 
accessed  publications  can  be  provided  on  a monthly  basis.  We  believe 
that  our  reports  comply  with  the  International  Coalition  of  Library 
Consortia  guidelines  - we  can  provide  reporting  on  our  sole  database,  by 
each  set  of  IP  addresses,  by  consortium,  by  username  (where 
applicable),  and  on  a monthly  basis.  Please  note  that  those  institutions 
using  IP  addresses  for  authentication  to  Electric  Library  would  be 
unable  to  obtain  reports  of  individual  user  activity  (institutions 
authenticating  via  Usemame/Password  would  be  able  to  retrieve  this 
information,  however). 

Facts  on  File  (Spring  2000) 

We  can  offer  statistical  reporting  by  IP  address  or  password  to  our 
subscribers.  Please  contact  Ben  Jacobs  (see  12  below)  to  discuss 
specific  needs. 

Gale  Group  (Spring  2000) 

InfoTrac  and  GaleNet  statistics  are  in  partial  compliment  with  ICOLC 
statistical  reporting  requirements.  The  Gale  Group  is  undertaking  a 
project  to  consolidate  the  separate  statistical  programs  and  incorporate 
full  ICOLC  compliance.  Statistics  will  be  generated  for  each  of  the 
various  subscribing  libraries.  Statistical  reports  include:  internal  and 
external  use  by  database  with  the  number  of  logins,  total  connect  time, 
average  login  time,  number  of  searches,  hits,  marked  citations,  emailed 
and  printed  citations  and  full  text.  The  journal  list  report  shows  the 
number  of  on-screen  views  and  retrievals,  with  a breakout  of  full  text 
retrievals.  Time  of  day  statistics  detail  number  of  users  by  time  of  day 
and  day  of  week  for  the  month  or  date  range  selected.  Statistics  can  be 
delivered  in  either  ASCII,  Flat  file  (i.e.  csv),  or  Postscript  formats.  In 
addition,  the  consortia  can  choose  to  receive  one  aggregate  report  for  a 
number  of  sites  and  can  also  have  individual  usage  reports  sent  to  each 
library  with  the  InfoTrac  Web  service.  The  reports  list  each  database 
subscribed  to  by  the  consortia  and/or  location  receiving  InfoTrac  Web 
services.  The  reports  show  the  number  of  Sessions,  On-Screen  Views 
and  Retrievals  (e.g.  local  printing,  email,  PDF  download.)  The  journal 
list  report  shows  the  number  of  on-screen  views  and  retrievals,  with  a 
breakout  of  full  text  retrievals.  Special  statistical  reports  can  be 
requested  through  Technical  Support  and  delivered  within  two  business 
days  of  the  request.  GaleNet  Database  Statistics  GaleNet  offers  an 
online  usage  reporting  service  to  its  customers  upon  request.  This 
database,  updated  on  a monthly  basis,  details  the  usage  by  individual 
site  (library,  school,  etc.)  and/or  the  consortia  as  a whole,  then  by 
database  and  by  type  of  usage  (database  access,  image,  other).  We  also 
have  the  ability  to  provide  more  detailed  reports  that  cover  number  of 
sessions,  average  number  of  users,  daily  usage  patterns,  etc. 

Grolier  (Spring  2000) 

User  statistics  for  each  account  will  be  reported  on  a password-protected 
Web  site  available  to  system  administrators  and  other  qualified 
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personnel  designated  by  the  customer.  Reports  are  available  for  IP 
address  ranges  and  referring  URLs  only,  not  for  "cookie"-authenticated 
workstations,  and  are  updated  monthly.  Accessibility  of  IP  addresses 
behind  proxy  servers  (subnet  level)  is  subject  to  technical  limitations  to 
be  determined  on  a case-by-case  basis.  Fields  reported  for  each 
institutionally  defined  set  of  IP  addresses  include  total  requests,  requests 
per  day,  number  of  files  requested,  hosts  served,  and  amount  of  data 
transferred.  Summaries  by  month,  day,  and  time  of  day  are  also 
provided.  Grolier  agrees  to  the  privacy  and  confidentiality  paragraphs 
(nos.  2 and  3)  in  the  ICOLC  "Guidelines  for  Statistical  Measures  of 
Usage  of  Web-Based  Indexed,  Abstracted,  and  Full-Text  Resources." 

Grove's  (Fall  1999) 

Both  of  the  Grove  online  products  were  launched  recently  and  our  usage 
statistics  reports  are  still  in  development.  Our  intention  is  to  follow  the 
guidelines  established  by  the  ICOLC  as  far  as  these  are  applicable  to  our 
products. 

Hoover's  (Fall  1999) 

We  provide  statistics  on  a request  basis.  Usage  reports  will  provide 
daily  statistics  on  usage  on  an  institutional  level  only.  This  complies  in 
part  with  the  ICOLC. 

HRAF  (Spring  2000) 

At  the  current  time  we  have  statistical  information  of  use  by  month  and 
by  IP  address.  However,  it  is  not  aggregated  by  the  institution  or  by  the 
consortium.  We  are  asking  our  provider  at  the  University  of  Michigan 
to  provide  more  detailed  statistical  information  to  comply  with  the 
guidelines  developed  by  the  ICOLC. 

INET  Library  (Spring  2000) 

Statistical  information  is  gathered  via  our  web  based  monitoring 
software.  We  gather  bit  ratios  based  on  IP  addressed  or  user  machines. 
Number  of  visits  are  user  specific  per  facility.  Additional  statistical 
information  can  be  developed  based  on  institutional  needs,  i.e. error 
recording  per  a specific  IP  address  or  site  usage  per  specific  IP  address. 
With  our  priority  service  we  can  provide  a significant  increase  in 
statistical  information  based  on  site  visits,  total  hit  ratios,  and  total  files 
utilized.  This  would  be  developed  specific  to  each  IP  address.  This 
statistical  reporting  process  complies  in  part  with  the  guidelines 
developed  by  the  ICOLC.  In  ICOLC’s  guidelines  it  states  that  "ICLOC 
is  preparing  a separate  guideline  on  Technical  Performance  of  web- 
based  parameters  (e.g.  downtime,  responsetime)."  Inventive 
Communications  will  strive  to  become  100%  compliant  to  these  new 
guidelines  once  ICOLC  publishes  its  version. 

Information  Quest  (Fall  1999) 

We  provide  usage  statistics  to  our  customers  that  are  currently  available 
on  a quarterly  basis.  These  statistics  state  the  number  of  logins,  the 
number  of  searches,  purchases,  online  hits  and  much  more!  To 
determine  if  we  meet  the  guidelines  developed  by  the  ICOLC,  please 
contact  !karle(S),eiq.com  <mailto:lkarle(a),eiq.com>  who  would  be  happy 
to  discuss  this. 

InfoUSA  (Fall  1999) 

Our  statistics  provide  each  library  how  many  searches,  how  many 
downloads  & how  many  downloaded  records.  This  can  be  a monthly, 
quarterly,  semi-annual  or  annual  pull.  It  doesn’t  look  like  our  statistics 
are  like  the  International  Coalition  of  Library  Consortia. 

Lexis-Nexis  (Spring  2000) 

We  will  be  providing  web  based  use  statistics  which  comply  with  the 
ICOLC  guidelines. 

Lightspan  (Fall  1999) 

Lightspan  can  provide  statistical  analysis  of  usage  of  the  Lightspan 
Network  either  at  the  school  level  or  by  password.  In  other  words,  we 
can  provide  analysis  of  overall  log-on  activity  of  the  Lightspan  Network 
to  determine  overall  rate  of  usage,  or  analysis  based  on  usage  by 
password.  This  second  level  of  analysis  is  useful  in  determining  rate  of 
usage  among  teachers,  students,  or  families  to  get  a sense  of  who  is 
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finding  value  in  the  Lightspan  Network  and  how  it  is  being  used. 

McGraw  Hill  (Spring  2000) 

Statistics  for  use  by  an  institution  will  be  provided  starting  in  July  of 
2000.  The  statistics  will  be  reported  daily  on  a website  accessible  only 
by  the  subscribing  institutions'  librarian  contact,  aggregated  weekly  and 
month.  All  efforts  are  being  made  to  comply  with  the  ICOLCs 
guidelines,  but  may  vary  to  the  extent  that  our  programming  allows  us 
to  capture  specific  information. 

NewsBank  (Fall  1999)  NewsBank 
(cont.) 

NewsBank  complies  with  the  International  Coalition  of  Library 
Consortia  (ICOLC)  in  the  following  ways:  NewsBank  can  provide 
usage  statistics  on  the  following  use  elements: 

• Number  of  Searches  (Keyword  and  Customized) 

• Number  of  Sessions  (Logins) 

• Number  of  Articles  Viewed 

Each  use  element  can  be  delineated  by  the  following  subdivisions: 

• By  each  specific  database  (Product) 

• By  each  institutionally-defined  set  of  IP  addresses 

• By  total  consortium 

• By  special  data  element  passed  by  subscriber  (etc.,  Account  or 
ID  Number) 

Note:  Usage  statistics  are  reported  daily  and  this  data  can  be  maintained 
for  24  months 

NICEM  (Spring  2000) 

We  can  provide  a monthly  report  of  the  number  of  records  that  result 
from  a search.  The  statistical  count  is  for  each  user  name. 

NoveList  (Fall  1999) 

NoveList  provides  statistics  on  a monthly  basis.  Statistics  are  available 
at  the  individual  site  level  and  the  consortial  level  based  on  IP  address. 
For  example,  building  level  statistics  are  available  for  buildings  that 
have  indicated  a unique  set  of  IP  addresses.  Information  is  provided  for 
the  following  elements: 

• Number  of  queries. 

• Number  of  sessions. 

• Number  of  records  viewed. 

Sites  which  provide  NoveList  with  a contact  person  and  e-mail  address, 
will  have  monthly  usage  statistics  e-mailed  to  them. 

OCLC  (Spring  2000) 
OCLC  (cont.) 

OCLC  provides  the  following  statistical  reports  for  the  FirstSearch 
service:  FirstSearch  Usage  Reports  OCLC  tracks  FirstSearch  usage 
statistics  and  provides  access  to  monthly  usage  reports  via  the  Web. 
Libraries  can  obtain  these  reports  at  anytime  from  the  Web 
(<http://www.stats.oclc.ore/>).  Libraries  are  provided  with  the  last 
month's  usage  as  well  as  the  past  twelve  months  of  usage  for  a rolling 
1 3 -months  of  usage  history.  Session-level  reports  are  available  for 
viewing  on  the  Web  on  the  same  1 3-month,  rolling  schedule. 

Overview  reports  include  6 months  of  data.  Session-level  reports 
include:  Overview  reports.  These  reports  provide  data  on  system  use 
such  as: 

• the  number  of  FirstSearch  authorizations  used  by  an  institution 

• the  number  of  sessions  or  the  number  of  FirstSearch  logons  by 
users 

• the  number  of  session  tumaways 

• the  total  number  of  searches  reported  in  a given  month  for  an 
institution 

Database  summary  reports.  These  reports  provide  data  on  system  usage 
at  the  database  level: 

• the  number  of  logons 

• the  number  of  searches 
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Some  of  the  information  available  from  the  online  statistics  includes: 
Consortium  Overview  This  report  shows  the  total  number  of 
authorizations  the  consortium  has  assigned,  the  number  of  times  users 
logged  on,  and  the  total  number  of  searches  conducted  and  the  total 
number  of  full-text  documents  ordered  over  a one-month  period. 
Institution  Overview  This  report  shows  the  maximum  number  of  logons 
used,  the  average  number  of  logons  used  and  the  total  number  of 
searches  conducted  over  a one-month  period.  It  also  shows  the  total 
number  of  users  who  logged  on  FirstSearch  over  a one-month  period. 
Institution  Activity  This  report  shows  the  number  of  times  users  logged 
in  each  database,  the  total  number  of  searches  conducted  in  each 
database  and  the  percentage  of  searches  conducted  in  each  database.  It 
also  shows  the  average  number  of  users  logged  into  a particular 
database  as  well  as  the  maximum  number  logged  into  a particular 
database  at  one  time. 

Full-Text  Activity  This  report  shows,  by  database  and  as  a total,  the 
number  of  ASCII  full-text  documents  that  users  viewed,  e-mailed  or 
printed  over  a one-month  period.  OCLC  at  present  is  in  partial 
compliance  with  the  ICOLC  guidelines.  We  are  planning  to  make 
appropriate  changes  to  our  statistical  reports  to  become  fully  compliant. 

OVID  (Spring  2000) 

OVID  is  in  compliance  with  the  following  items  in  that  they  provide 
statistics  by: 

• Each  specific  database  of  the  provider 

• Each  institutionally-defined  set  of  IP  addresses  / locators  to 
subnet  level 

• Total  consortium 

Note:  Assuming  that  consortium  members  are  assigned  distinct 
group  ID.  In  consortiums  this  is  normally  the  case 

• Time  period 

• Number  of  queries  (Searches)  categorized  as  appropriate  for 
the  vendor’s  information 

• Number  of  Menu  Selections  categorized  as  appropriate  to  the 
vendor’s  system 

Note:  Assuming  that  appropriate  data  would  be  display  record, 
view  record,  print  record  etc... 

• Number  of  sessions  (Logins),  if  relevant,  must  be  provided  as  a 
measure  of  simultaneous  use 

Note:  Yes,  but  not  simultaneous  use  Number  of  tum-always,  if 
relevant,  as  a contract  limit  (e.g.,  requests  exceed  simultaneous 
user  limit) 

• Number  of  items  examined  by  citations  displayed 

• Number  of  items  examined  by  full  text  displayed  broken  down 
by  title,  ISSN  with  title  listed,  or  other  title  identifier  as 
appropriate 

OVID  is  not  in  compliance  with  the  following  items: 

• Number  of  items  examined  by  special  data  element  passed  by 
subscriber  (e.g.,  account  or  ID  number) 

Note:  Not  at  this  time  in  the  statistics  program  but  we  do 
collect  this  information  in  the  stats  file  if  it  is  passed  through  as 
part  of  a jumpstart  command 

Primary  Source  Media  (Fall  1999) 

Primary  Source  Media  can  provide  user  statistics  on  a predefined  or  as- 
needed  basis,  utilizing  the  Webtrends  software  product  for  this  purpose. 
Most  of  the  guidelines  developed  by  ICOLC  are  addressed  by  our 
statistical  reporting  capabilities. 

Roth  (Fall  1999) 

We  provide  usage  statistics  at  any  given  intervals  for  libraries  (normally 
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on  a quarterly  basis).  Our  reports  conform  to  the  International  Coalition 
of  Library  Consortia  guidelines. 

Silver  Platter  (Spring  2000) 

We  provide  statistics  that  are  e-mailed  on  a monthly  basis  for  usage. 

The  usage  statistics  include  the  database  used,  time,  # of  logins  and  # of 
records  retrieved. 

SIRS  (Spring  2000)  SIRS  (cont.) 

SIRS  Usage  Statistics  were  originally  constructed  using  the  Guidelines 
for  Statistical  Measures  of  Usage  of  Web-Based  Indexed.  Abstracted 

and  Full  Text  Resources  developed  by  the  International  Coalition  of 
Library  Consortia  (ICOLC).  They  comply  with  these  guidelines.  SIRS 
provides  statistics  on  a monthly  basis.  They  are  e-mailed  each  month  to 
a designated  contact.  The  statistics  include: 

SIRS  Knowledge  Source 

® total  accesses  to  all  databases 

• total  sessions 

• total  number  of  references  viewed 

• total  number  of  searches 

• total  for  each  type  of  search 

• total  accesses  to  almanac  databases 

• total  number  of  summaries/descriptors  viewed 

• total  number  of  sources  viewed 

• total  number  of  graphics  viewed 

• total  number  of  article  e-mailed 

• total  accesses  to  the  Search  History 

• total  accesses  to  the  Tagged  List 

• total  accesses  to  the  Dictionary/Thesaurus 

• total  accesses  to  the  Help  & Tips. 

SIRS  Discoverer  Deluxe 

• total  accesses 

• total  number  of  reference  materials  viewed 

• total  number  of  searches 

• total  number  of  searches  by  type  of  search 

• total  number  of  full-text  articles  viewed 

• total  number  of  World  Almanac  for  Kids  keyword  searches 

• total  number  of  World  Almanac  for  Kids  table  of  content 

searches 

• total  number  of  encyclopedia  searches 

• total  number  of  Dictionary/Thesaurus  searches 

• total  number  of  sources/summaries  viewed 

• total  number  of  articles  e-mailed 

• total  number  of  charts  viewed 

• total  number  of  graphics  viewed 

• total  accesses  to  Workbooks 

Softline  (Spring  2000) 

SLl’s  provision  of  usage  statistics  to  subscribing  libraries  complies 
where  appropriate  with  the  ICOLC  guidelines.  SLI  will  provide 
libraries  with  usage  statistics  which  include  the  following  information  : 
No.  of  sessions;  No.  of  searches;  No.  of  articles  retrieved;  No.  of 
publications  retrieved;  Maximum  no.  of  concurrent  users;  No.  of  times 
access  is  denied.  Statistics  will  be  distributed  by  e-mail  to  an  address 
designated  by  the  library,  on  a monthly  or  quarterly  basis,  as  requested. 
SLI  is  also  planning  development  of  access  to  usage  statistics  via  a 
website. 

Standard  & Poor's  (Fall  1999) 

We  are  currently  creating  user  statistics  that  comply  with  internal 
requirements  established  by  Standard  & Poor’s.  Further  review  of  the 
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ICOLC  is  necessary  to  determine  if  our  statistical  reporting  complies. 

Teton  Data  Systems’  STATfRef  (Spring 
2000) 

Our  intranet  product  which  resides  on  your  own  web  server  within  your 
environment  has  its  own  administration  page  that  will  provide  reporting 
capabilities  based  on  a specific  date  range.  Users  can  obtain 
information  on  anyone  who  has  been  denied  access. 

H.W.  Wilson  (Spring  2000) 

Statistical  reporting  complies  in  part  with  the  guidelines  developed  by 
the  International  Coalition  of  Library  Consortia.  The  current 
Wilson  Web  report  includes  the  name  of  the  institution,  number  of 
searches  and  number  of  records  downloaded,  by  database,  for  the 
current  month,  last  month,  and  year-to-date.  Reports  can  be  produced 
by  databases;  by  community  of  interest  entity-defined  IP  addresses;  by 
community  of  interest  type  and  by  total  of  all  community  of  interest 
types;  by  special  element  passed  by  community  of  interest  entity;  and 
by  time  period.  Presently  the  available  reports  are  generated  monthly 
by  the  company  and  provided  to  customers  in  hard-copy  format  or  via  e- 
mail  upon  request.  We  are  also  evaluating  the  most  efficient  manner  by 
which  to  make  the  reports  available  in  electronic  format,  WilsonWeb 
reports  are  in  a continued  state  of  development.  We  are  using 
requirements  of  our  customers  as  the  basis  for  enhancements  to  the 
reporting  functionality. 

WorldBook  (Fall  1999) 

Currently  we  are  not  providing  this  information,  however  after  the  first 
of  the  year  the  company  will  begin  to  provide  some  data.  Sorry  at  this 
time  we  are  not  sure  what  that  data  will  be. 

This  data  was  collected  by  staff  at  the  Washington  State  Library  in  Fall  1999  and  Spring  2000,  in  conjunction  with 
Statewide  Database  Trials.  It  was  compiled  from  responses  supplied  by  vendors  and  used  as  a handout  at  a program 
on  Capturing  and  Using  Statistics  about  Electronic  Resources,  presented  at  the  Washington  Library  Association 
annual  conference,  May  17-19,  2000  in  Tacoma,  Washington. 

If  you  have  questions  about  this  document,  contact  Jeanne  Crisp  at  the  Washington  State  Library,  360-704-5255  or 
icrisp@statelib.wa.gov. 
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Appendix  E.  U.  Pennsylvania  Library  Reports 


Following  pages  contain  sample  reports  obtained  from  U.  Penn.  Library. 

What  We  Can  Measure  Regarding  the  Use  of  Library  Databases 

A matrix  of  licensed  vendor  databases  in  terms  of  the  reported  statistics. 

Use  of  Licensed  Databases,  Measured  in  Attempted  Log-Ons.  November  1999-June  2000 

The  table  shows  the  number  of  "attempted  log-ons"  recorded  by  user  click-throughs  on  the 
library's  database  web  site.  The  raw  data  are  tracked  by  the  originating  IP  (Internet)  addresses 
and  the  totals  are  calculated  for  each  school  on  campus,  library  buildings,  and  the  modem 
pool. 

Heavily-  & Lesser-Used  Databases,  Measured  in  Attempted  Log-Ons.  November  1999-June  2000 

This  table  that  shows  the  most  heavily  used  licensed  databases  by  the  number  of  attempted 
log-ons.  The  statistics  are  allocated  to  schools,  library,  and  to  the  modem  pool. 

Demand  for  Licensed  Resources.  Attempted  logons  averaged  by  time  of  day.  November  1999- 
June  2000 


The  graphs  shows  the  intensity  of  licensed  database  use  by  time  of  day. 

Demand  for  Licensed  Resources.  Hourly  attempted  logons  averaged  by  Major  Network  Domains, 
November  1999-June  2000 


The  graphs  shows  the  intensity  of  licensed  database  by  location  (where  the  requests  were 
made)  and  by  time  of  day. 


What  We  Can  Measure  Regarding  the  Use  of  Library  Databases 

updated  7/2000 

Available  Measures 


Database  Xy  Client 

Source 

Ct* 

new 
in  00 
n 

Amico  Library 

X 

Web;  Telnet 

RLIN/Eureka 

1 

Anthropological  Literature 

X 

Web;  Telnet 

RLIN/Eureka 

2 

Avery  Index 

X 

Web;  Telnet 

RLIN/Eureka 

3 

Bibliography  of  the  History  of  Art 

X 

Web;  Telnet 

RLIN/Eureka 

4 

Blackwell’s  Table  of  Contents 

X 

Web;  Telnet 

RLIN/Eureka 

5 

Chicano  Database 

X 

Web;  Telnet 

RLIN/Eureka 

6 

ESTC 

X 

Web;  Telnet 

RLIN/Eureka 

7 

Francis 

X 

Web;  Telnet 

RLIN/Eureka 

8 

Hispanic  American  Periodicals  Index 

X 

Web;  Telnet 

RLIN/Eureka 

9 

History  of  Science  and  Technology 

X 

Web;  Telnet 

RLIN/Eureka 

10 

Index  to  19th  Cent  American  Art  Period 

X 

Web;  Telnet 

RLIN/Eureka 

11 

n 

Inside  Information  Plus 

X 

Web;  Telnet 

RLIN/Eureka 

12 

n 

RLIN  Bib  file 

X 

Web;  Telnet 

RLIN/Eureka 

13 

World  Law  Index 

X 

Web;  Telnet 

RLIN/Eureka 

14 

n 

Index  to  Foreign  Legal  Periodicals 

X 

Web;  Telnet 

RLIN/Eureka 

15 

n 

Scipio 

X 

Web;  Telnet 

RLIN/Eureka 

16 

PROMT 

X 

X 

Web;  Telnet 

IAC 

17 

Academic  Index 

X 

X 

Web;  Telnet 

IAC 

18 

General  Business  File 

X 

X 

Web;  Telnet 

IAC 

19 

Social  Sciences  Citation  Index 

X 

avrg 

X 

Web 

ISI 

20 

Science  Citation  Index 

X 

avrg 

X 

Web 

ISI 

21 

Arts  & Humanities  Citation  Index 

X 

avrg 

X 

Web 

ISI 

22 

Journals  OOvid  Full-text 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

23 

Sociological  Abstracts  remotely  loaded  -r 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

24 

PsycINFO  & Psyclnfo  Hist.  File 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

25 

MEDLINE 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

26 

INSPEC  r 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

27 

Health  & Psychosocial  Instruments  r 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

28 

HealthSTAR  r 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

29 

ERIC  r 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

30 

EconLit  r 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

31 

CINAHL 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

32 

CancerLit 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

33 

CAB  Abstracts  r 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

34 

BIOS  IS  Previews 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

35 

BioethicsLine 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

36 

Art  Index 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

37 

Applied  Science  & Technology 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

38 

AIDSLINE 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

39 

Agricola  r 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

40 

Ageline  r 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

41 

ABI/Inform 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

42 

Cochrane  Database  of  Systemic  Revs. 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

43 

n 

Best  Evidence 

X 

X 

X 

X 

Web;  Telnet;  Java 

Ovid 

44 

n 

World  Cat 

X 

X 

Web;  Telnet 

OCLC 

45 

PAIS  International 

X 

X 

X 

X 

Web;  Telnet;  Java 

OCLC 

46 

RILM  Abstracts  of  Music  Literature 

X 

X 

X 

X 

Web;  Telnet;  Java 

OCLC 

47 

MLA  Int'IBibliography 

X 

X 

X 

X 

Web;  Telnet;  Java 

OCLC 

48 

GeoRef 

X 

X 

Web;  Telnet 

OCLC 

49 

Books  In  Print 

X 

X 

Web;  Telnet 

OCLC 

50 

ATLA  Religion  Database 

X 

X 

Web;  Telnet 

OCLC 

51 

Philasophers  Index 

X 

X 

Web 

S Ivr  Platter 

52 

n 

El  Compendex 

X 

X 

Web 

Engineering  Info  Inc 

53 

EIU’s  Business  Newsletter... 

X 

X 

Web 

Slvr  Platter 

54 

n 

EIU  Investing,  Licensing  and  Trading 

- 

- 

- 

- 

- 

* 

Web 

Silverplatter 

55 

n 

EIU  Views 

X 

X 

Web 

S Ivr  Platter 

56 

n 

Intern'l  Bibliography  of  the  Social  Scis. 

X 

X 

Web 

S Ivr  Platter 

57 

n 

International  Political  Science  Absts. 

X 

X 

Web 

Slvr  Platter 

58 

n 

LLBA  Abstracts 

X 

X 

Web 

S Ivr  Platter 

59 

Social  Work  Abstracts 

X 

X 

Web 

S Ivr  Platter 

60 

OED 

X 

Web 

Local 

61 

Britannica 

X 

X 

Web 

BOL 

62 

Beilstein  via  Crossfire 

X 

Win/MAC 

Beilstein 

63 

Academic  Universe  (Lexis/Nexis  for  the  web) 

X 

X 

Web 

Reed/E  Isv 

64 

SciFinder  Scholar 

X 

Win/MAC 

Chem  Abs 

65 
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Database  Client 

Source 

Ct. 

New 
in  99 

Franklin  (OPAC) 

X 

Web:Telnet 

Local 

.... 

Ethnic  Newswatch 

X 

X 

Web 

Softline  Info 

66 

n 

GenderWatch 

X 

X 

Web 

Softline  Info 

67 

n 

Periodicals  Contents  Index 

X 

Web 

Chad.H 

68 

Patrologia  Latina 

X 

Web 

Chad.H 

69 

Literature  Online 

X 

Web 

Chad.H 

70 

Int'l  Index  to  Music  Perdcls 

X 

Web 

Chad.H 

71 

African  American  Biographical  Dbase 

X 

Web 

Chad.H 

72 

n 

Palmer  Index  to  the  London  Times 

X 

Web 

Chad.H 

73 

n 

Historical  Index  to  the  NYTimes 

X 

Web 

Chad.H 

74 

n 

America:  History  and  Life 

- 

- 

X 

- 

- 

- 

Web 

ABC-CLIO 

75 

n 

Art  Bibliographies  Modern 

- 

- 

X 

- 

- 

- 

Web 

ABC-CLIO 

76 

n 

Historical  Abstracts 

- 

- 

X 

- 

- 

- 

Web 

ABC-CLIO 

77 

n 

MathSciNet 

X 

X 

Web 

AMS 

78 

Databases  with  no  use  measures 

ABI/Inform/proquest  (ovid-count  once) 

- 

- 

- 

- 

- 

- 

web 

Proquest 

ABSEES  (Slavic  & E.  Eur.  Stds) 

- 

- 

- 

- 

- 

- 

Web;  Telnet 

AAASS 

79 

Access  UN 

- 

- 

- 

- 

- 

- 

Web 

Newsbank 

80 

Access  UN DB 

- 

- 

- 

- 

- 

- 

Web 

Newsbank 

81 

N 

Accessible  Archives 

- 

- 

- 

- 

- 

- 

Web 

AccArch 

82 

n 

Anthropological  Index  * 

- 

- 

- 

- 

- 

- 

Web 

Free 

83 

AP  Photo  Archive 

- 

- 

- 

- 

- 

- 

Web 

Accunet-AP 

84 

n 

ARL  Latin  Americanist  Research  Resources 

- 

1 

- 

- 

- 

- 

Web 

ARL 

85 

n 

ARTFL 

- 

- 

- 

- 

- 

- 

Web 

U Chicago 

86 

BankScope 

- 

- 

- 

- 

- 

- 

Web 

Bureau  Van  Dijk 

87 

n 

Bibliography  of  Asian  Studies 

- 

- 

- 

- 

- 

- 

Web 

Assc  Asian  St 

88 

n 

Business  & Industry 

- 

- 

- 

- 

- 

- 

Web 

Rspnsv  db 

89 

CCH  Commerce  Clearinghouse  Tax  Res 

- 

- 

- 

- 

- 

- 

Web 

Commc  Clearh 

90 

n 

CIAO 

- 

- 

- 

- 

- 

- 

Web 

Columbia  U 

91 

ClOS/Comserv 

- 

- 

- 

- 

- 

- 

Web 

Comserv 

92 

n 

CogNet 

- 

- 

- 

- 

- 

- 

Web 

MIT  Press 

93 

N 

Congressional  Universe 

- 

- 

- 

- 

- 

- 

Web 

CIS 

94 

Contemporary  Women's  Issues 

- 

- 

- 

- 

- 

- 

Web 

Rspnsv  db 

95 

Current  Index  to  Statistics 

- 

- 

- 

- 

- 

- 

Web 

CIS 

96 

n 

D&B  Million  Dollar  Database 

- 

- 

- 

- 

- 

- 

Web 

Dun  /Bradstreet 

97 

n 

Declassifiedm  Documents  Online 

- 

- 

- 

- 

- 

- 

Web 

Gale 

98 

n 

Dissertation  Abstracts 

- 

- 

- 

- 

- 

- 

Web 

UMI 

99 

n 

Dow  Jones  Interactive 

- 

- 

- 

- 

- 

- 

Web 

Dow  Jones 

100 

Early  English  Books  Online 

- 

- 

- 

- 

- 

- 

web 

proquest 

101 

n 

EdgarScan 

- 

- 

- 

- 

- 

- 

web 

free 

102 

n 

EIU  Country  Commerce 

- 

- 

- 

- 

- 

- 

web 

Econist  Intel  Unit 

103 

N 

EIU  Investing,  Licensing  and  Trading 

- 

- 

- 

- 

- 

- 

Web 

Silverplatter 

104 

n 

Encyclopedia  of  Assns 

- 

- 

- 

- 

- 

- 

Web 

Gale 

105 

Euromonitor's  Global  Market  Information 

- 

- 

- 

- 

- 

- 

Web 

Euromarket  PLC 

106 

n 

Facts  on  File 

- 

- 

- 

- 

- 

- 

Web 

107 

n 

Gale  Business  Resources 

- 

- 

- 

- 

- 

- 

Web 

Gale 

108 

n 

Gartner  Intraweb 

- 

- 

- 

- 

- 

- 

Web 

Gartner  grp 

109 

n 

Global  Access 

- 

- 

- 

- 

- 

- 

Web 

Disclosure 

110 

n 

Grove  Dictionary  of  Art  Online 

- 

- 

- 

- 

- 

- 

Web 

Grove 

111 

n 

Grover  Dictionary  of  Opera 

. 

- 

- 

- 

- 

- 

Web 

Grove 

112 

n 

Harp  Week 

- 

♦ 

- 

- 

- 

- 

Web 

Harpweek 

113 

n 

Hbook  of  Latin  American  Studies 

- 

- 

- 

- 

- 

- 

Web 

Free 

114 

HISTLINE 

- 

- 

- 

- 

- 

- 

Web 

NLM 

115 

n 

Hoover's  Online 

- 

- 

- 

- 

- 

- 

Web 

Hoover's.com 

116 

n 

eHRAF  Collection  of  Archaeology 

- 

- 

- 

- 

- 

- 

Web 

Umich 

117 

n 

eHRAF  Collection  of  Ethnography 

- 

- 

- 

- 

- 

- 

Web 

Umich 

118 

n 

ICM  Online 

* 

- 

- 

- 

- 

- 

Web 

ICM  Conferences 

119 

N 

114 
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Available  Measures 


Database  / Client 

Source 

Ct. 

New 
in  99 

Index  of  Christian  Art 

- 

- 

- 

- 

- 

- 

Web 

Princeton  U 

120 

n 

Index  to  Early  American  Periodicals 

- 

- 

- 

- 

- 

- 

Web 

Computer  Index  Sys 

121 

n 

ISI  Emerging  Markets 

- 

- 

- 

- 

- 

- 

Web 

Intrn’l  Securities 

122 

n 

ITER:Gateway  to  the  Renaissance 

- 

- 

- 

- 

- 

- 

Web 

Ren  Soc,etal 

123 

n 

Johns  Hopkins  Guide  to  Lit  Theory 

- 

- 

- 

- 

- 

- 

Web 

JH  U Press 

124 

Knowledge  at  Wharton 

- 

- 

- 

- 

- 

- 

Web 

125 

N 

Mental  Measurements  Yearbook 

- 

- 

- 

- 

- 

- 

Web 

Slvr  Platter 

126 

n 

Middle  English  Compendium 

- 

- 

- 

- 

- 

- 

Web 

UMICH 

127 

n 

MITECS 

- 

- 

- 

- 

- 

- 

Web 

MIT 

128 

n 

Monthly  Catalog 

_ * 

- 

- 

- 

- 

Web;  Telnet 

Free 

129 

Multex 

fW.j 

^:!:l 

- 

- 

- 

- 

Web 

130 

n 

Music  Index 

- 

- 

- 

- 

- 

. 

Web 

Harmony  park 

131 

n 

NBER  Working  Papers 

- 

- 

- 

- 

- 

- 

Web 

132 

NCJRS  Abstracts  Database 

- 

- 

- 

- 

- 

- 

Web 

Free 

133 

n 

NetEC 

- 

- 

- 

- 

- 

Web 

Free 

134 

n 

Old  English  Corpus 

- 

- 

- 

- 

- 

- 

Web 

UMICH 

135 

n 

Poiesis 

- 

- 

- 

- 

- 

- 

Web 

Phil  doc  centr 

136 

n 

Polling  the  Nations 

- 

- 

- 

- 

- 

. 

Web 

ORS  Publishing 

137 

n 

Population  Index  (unrestricted) 

- 

- 

- 

- 

- 

. 

Web 

Free 

138 

Pblctns  & Broadcast  Media 

- 

- 

- 

- 

Web 

Gale 

139 

RAMBI 

- 

- 

- 

- 

- 

- 

Web 

Free 

140 

Research  Bank  Web-Investext 

- 

- 

- 

- 

- 

- 

Web 

Investext 

141 

n 

Russian  Academy  of  Sciences  Bibliography 

- 

- 

- 

- 

- 

Web 

RAS 

142 

n 

Social  Science  Electronic  Data  Library 

- 

- 

- 

- 

- 

- 

Web 

Sociometrics  corp 

143 

n 

Sports  Business  Research  Network 

- 

- 

- 

- 

- 

- 

Web 

Sprt  Bus  RN 

144 

State  Capital  Universe 

- 

- 

- 

- 

- 

- 

Web 

CIS 

145 

n 

Statistical  Universe 

- 

- 

- 

- 

- 

- 

Web 

CIS 

146 

n 

STAT-USA/Internet 

- 

- 

- 

- 

- 

- 

Web 

Dpt  of  Comm 

147 

n 

Table  Base 

- 

- 

- 

- 

- 

- 

Web 

Rspnsv  db 

148 

Times  Literary  Supplement 

- 

- 

- 

- 

- 

- 

Web 

London  Times 

149 

n 

United  Nations  Treaty  Collection 

- 

- 

- 

- 

- 

- 

Web 

UN 

150 

n 

Women  Writers  Online 

- 

- 

- 

- 

- 

- 

Web 

Brown  Univ 

151 

n 

Available  Measures 


E-Journal  Collections  <^/  <$)/  Client 

Source 

ct.  1 

ACS  Journals 

X 

X 

Web 

JSTOR 

X 

X 

X 

Web 

Johns  Hopkins  Univ  Press 

X 

Web 

Elec.  Collections  Online:  Journals 

X 

X 

X 

Web 

Academic  Ideal 

Web 

American  Institute  of  Physics 

Web 

American  Mathematical  Society 

Web 

Annual  Reviews 

Web 

Link  (Medicine) 

Web 

IEEE 

- 

- 

- 

- 

- 

- 

Web 

Elsevier  Scientific 

- 

- 

- 

- 

. 

- 

Web 

Cambridge  Scientific 

- 

- 

- 

- 

- 

- 

Web 

Springer 

. 

. 

. 

- 

- 

. 

Web 

Wiley 

- 

- 

- 

- 

- 

- 

Web 
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1,599 

1,575 

1,560 

1,525 

1,508 

1,506 
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1,327 
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1,066 
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1,016 
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Appendix  F 

Yale  Library's  Database  Usage  Statistics  Webpage 

The  screen  image  below  shows  the  initial  page  of  the  central  usage  statistics  webpage  maintained  for 
internal  use.  Vendor  supplied  statistics  are  accompanied  with  a description  of  report  delivery  and  graphs 
showing  usage  trend  over  time. 
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Appendix  G 

Responses  to  the  Organizational  Structure  Question 

The  following  question  was  sent  to  the  project  participants  early  October.  Twelve  responses  were 
collected  and  presented  below. 

What  kind  of  (official/ad  hoc)  organizational  structure  do  you  have  right  now  to  coordinate  the  collection 
and  analysis  of  data  about  electronic  resource  use?  Please  list  the  titles  of  key  personnel  involved  and 
their  roles. 


Titles  of  staff  members 

Duties  / roles 

Collection  Development  Team 

1)  Electronic  Resources  Coordinator 

2)  Collection  Development  Coordinators  for 
Humanities,  Social  Sciences,  and  Science  (3 
people) 

3)  A team  leader 

1)  Coordinating,  selecting,  licensing,  and 
implementing  e-resources.  Also,  compiling 
usage  data,  creating  reports,  and 
disseminating  information  selectors,  and 
other  stakeholders. 

2)  Coordinating  content  issues. 

1)  A serials  acquisitions  librarian 

2)  A systems  librarian 

3)  The  Assistant  Dean  for  collection  development 

1)  Reviewing  and  signing  licenses  and  often 
receiving  usage  data. 

2)  Reviewing  and  installing  software  designed 
to  capture  the  data. 

1)  Coordinator 

1)  Coordinating  data  collection  and  analyzing 
data. 

Fiscal  Services  & Data  Team 

- Tracking  their  acquisitions  expenditures  and 

the  cost  of  their  e-resources. 

- Analyzing  the  data 

- Generating  reports  and  reporting  templates. 

Networked  Services  Team 

1)  Networked  services  librarian 

2)  Acquisitions  librarian 

3)  Principal  bibliographer 

4)  Two  rotating  members 

5)  Graduate  Student  majoring  in  statistics 

2)  Negotiating  licenses 

5)  Compiling  usage  information  for  the  team  to 
review  in  making  renewal  decisions. 

1)  Head,  Business  Administration 

2)  A policy  quadrant  (?) 

3)  Bibliographers  in  collections  development 

2)  Coordinating  web  use  data. 

3)  Collecting  and  analyzing  the  data. 

1)  NETDOC  (NETworked  Databases  on  Campus) 
Programmer 

2)  NETDOC  Developer 

3)  Head  of  Serials  Acquisitions 

4)  Reference  Librarians 

1,  2,  3,  and  4)  Involving  in  data  collecting  and 
analyzing  the  statistics. 

1)  Coordinator  for  Electronic  Resources 

2)  Coordinator  for  Collection  Management 

3)  Head  of  Acquisitions 

4)  Graduate  Assistants 

1 , 2 and  3)  Coordinating  the  collection  and 
analysis  of  data  about  electronic  resources 
and  assisting  the  manager  of  Library  in  data 
analysis. 

4)  Compiling  the  available  usage  statistics  and 
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Titles  of  staff  members 

Duties  / roles 

5)  Manager,  MIS 

working  with  staff  to  make  renewal  decisions 
5)  Working  with  all  of  the  above  to  assist  in 
data  analysis. 

- User  Needs  Assessment  Committee  and  Associate 
Director  for  Assessment 

- Supporting  the  assessment  activities  of  the 
organization  and  clarifying  traditional 
measures  as  well  as  developing  new 
measures 

1)  Associate  Director 

2)  Coordinator 

3)  Database  Administrator 

1)  Director  for  collection  Management 

2)  Dean  of  library 

3)  Director  for  instruction 

4)  Director  for  Reference  Services 

5)  Director  for  Technical  Services 

6)  Web  Team 

7)  College  Librarians 

7)  Liaison  to  colleges  with  instruction  and 
collection  management  responsibilities 

1)  Electronic  Publishing  and  Collections  Librarian 
(Past) 

2)  Digital  Collections  Specialist  (Present) 

3)  Reference  Librarians 

4)  Student  assistants 

1)  Collating  and  process  and  make  available  all 
stats 

2)  Reporting  all  available  stats  to  Electronic 
publishing  and  Collections  Librarian' 

3)  Coordinating  Medical  library  statistics. 

4)  Processing  all  statistics 

Department  chairs  for  Automated  Systems  and 
Collection  Management  coordinate  and  analyze 
electronic  resource  use  data 

Coordinate  and  analyze  electronic  resource  use 
data 

1)  Coordinator  for  Electronic  Resources 

2)  Computer  Applications  Specialist 

3)  Web  Services  Librarian 

1)  Coordinates  the  collection  of  vendor- 
produced  statistics,  decides  what  statistics  to 
record  and  format  for  Web  delivery 

2)  Supervises  a student  who  does  the  data  entry 

3)  Coordinates  other  statistics  gathering  such  as 
log  analysis  and  click  through  counting 
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EXECUTIVE  SUMMARY 

There  is  a critical  need  for  research  libraries  to  develop  new  statistics  and  measures  to  describe 
network  services  and  resources.  This  study  provides  one  approach  for  describing  and  measuring  some  of 
the  resources,  uses,  and  expenditures  for  supporting  networked  services  in  a research  library  setting.  This 
manual  is,  nevertheless,  one  product  of  a larger  effort  to  develop  new  measures  and  evaluation  techniques 
by  the  Association  of  Research  Libraries  (ARL)  http://www.arl.org/stats/newmeas/newmeas.html.  The 
E-Metrics  project,  of  which  this  report  is  one  product,  began  in  April  2000,  and  is  scheduled  for 
completion  in  December  of  2001.  The  project  is  funded  by  a group  of  24  ARL  member  libraries.  Other 
key  products  from  the  E-Metrics  project  are  still  in  process. 

Based  on  a substantial  field-testing  process  (described  in  detail  in  the  report),  the  study  team 
recommends  a number  of  network  statistics  and  performance  measures  that  provide  indicators  of 
library  networked  services  and  resources.  The  statistics  include: 

• Patron  accessible  resources  - Number  of  electronic  full-text  journals,  Number  of  electronic 
reference  sources,  and  Number  of  electronic  books; 

• Use  of  electronic  resources  and  services  - Number  of  electronic  reference  transactions,  Number 
of  logins  (sessions)  to  electronic  databases,  Number  of  queries  (searches)  in  electronic  databases, 
Number  of  items  requested  in  electronic  databases,  and  Virtual  visits  to  library’s  website  and 
catalog; 

• Expenditures  for  networked  resources  and  related  infrastructure  - Cost  of  electronic  full-text 
journals,  Cost  of  electronic  reference  sources,  Cost  of  electronic  books,  Library  expenditures  for 
bibliographic  utilities,  networks,  and  consortia,  and  External  expenditures  for  bibliographic 
utilities,  networks,  and  consortia;  and 

• Library  digitization  activities  - Size  of  library  digital  collection,  Use  of  library  digital  collection, 
and  Cost  of  digital  collection  construction  and  management. 

The  performance  measures  are  composite  and/or  combinations  of  the  above  network  statistics  along 
with,  in  some  cases,  non-network  statistics  already  collected  by  ARL  libraries  (e.g.,  number  of  visitors  to 
the  library).  The  performance  measures  are: 

• Percentage  of  electronic  reference  transactions  of  total  reference  transactions; 

• Percentage  of  virtual  library  visits  of  all  library  visits;  and 

• Percentage  of  electronic  books  of  all  monographs. 

These  statistics  and  measures  will  provide  research  libraries  with  an  important  and  useful  set  of  tools 
to  describe  and  assess  network  resources  and  services.  The  manual  also  provides  libraries  with  guidance 
regarding  the  use  to  which  the  network  statistics  and  measures  can  be  put. 

The  report  and  manual  offered  here  has  a number  of  specific  goals  and  objectives.  Its  primary  goal  is 
to  provide  a beginning  approach  for  research  libraries  to  better  describe  the  use  and  users  of  their 
networked  services.  A secondary  goal  is  to  increase  the  visibility  and  importance  of  developing  such 
statistics  and  measures.  Specific  objectives  of  the  report  and  manual  are  to: 

• Identify  selected  key  statistics  and  measures  that  can  describe  use  and  users  of  electronic  and 
networked  services; 

• Standardize  procedures  and  definitions  to  collect  these  statistics  and  measures;  and 

• Increase  awareness  of  selected  issues  related  to  collecting,  analyzing,  and  reporting  the  data  to 
produce  these  statistics  and  measures. 
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The  statistics  and  measures  offered  here  will  need  to  be  continually  developed,  expanded,  refined, 
and  possibly  eliminated  over  time. 

A key  component  of  the  project  has  been  to  work  with  vendors  and  other  organizations 
regarding  the  collection,  manipulation,  and  reporting  of  vendor-supplied  online  database  data. 

Many  of  the  statistics  described  here  resulted  from  the  cooperative  efforts  among  these  vendors 
and  other  national/intemational  groups  interested  in  developing  such  statistics.  Such  efforts 
should  be  continued. 

The  report  also  discusses  developments  related  to  the  degree  to  which  research  libraries  are  able  to 
help  their  larger  institution  reach  its  goals  or  institutional  outcomes.  These  outcomes  may  be  articulated 
in  strategic  planning  documents,  in  conversations  with  the  provost  or  academic  deans,  as  part  of  regional 
accreditation  standards,  or  in  state  legislation.  Nonetheless,  ARL  libraries  need  to  develop  a process  to 
identify  and  operationalize  library  outcomes  that  contribute  to  institutional  outcomes.  Establishing  such  a 
process  allows  the  library  to  inform  key  stakeholders  in  the  university  of  the  library’s  role  in  institutional 
outcomes  and  insures  that  the  institutional  outcomes  to  which  the  library  has  (or  may  have)  links  are  in 
fact  appropriate.  While  progress  has  been  made  in  this  area  of  measurement,  much  work  remains  to  be 
done. 

There  are  a number  of  issues  and  challenges  that  will  affect  the  library’s  ability  to  collect  statistics 
and  measures  to  describe  its  electronic  resources  and  services.  These  issues  and  suggestions  for  how  the 
library  can  best  address  and  resolve  them  are  discussed  within  the  report.  Some  research  libraries  possess 
inadequate  resources,  staffing,  and  expertise  to  collect,  manage,  and  report  the  data  related  to  describing 
networked  services.  For  these  libraries,  some  organizational  development  and  commitment  to  collecting 
and  using  these  data  may  be  necessary  to  take  advantage  of  the  measurement  tools  and  techniques 
outlined  in  this  report. 

Given  the  rapidly  changing  technology  environment,  the  changing  milieu  of  higher  education, 
changing  organizational  structures  within  ARL  libraries,  and  the  complexity  of  measuring  such 
networked  services,  it  is  almost  certain  that  the  statistics  and  measures  proposed  in  this  study  will 
continue  to  evolve.  The  measurement  tools  offered  in  this  report,  however,  will  provide  research 
librarians  with  important  techniques  to  count,  describe,  and  report  networked  services  and  resources  in 
their  libraries. 
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HOW  TO  USE  THIS  REPORT 


This  is  an  extensive  report  in  terms  of  length  and  topics  discussed.  Although  you 
are  encouraged  to  read  the  entire  report,  the  information  below  is  offered  to  help  you 
locate  those  sections  you  find  most  useful. 

The  report  contains  the  following: 

• The  Project  Overview  (pp.  xi-xx)  summarizes  the  information  in  parts  1,  2,  3, 
and  4; 

• Part  1 (pp.  1-26)  includes  the  context  and  issues  for  the  project; 

• Part  2 (pp.  27-42)  discusses  library  statistics  field-testing; 

• Part  3 (pp.  43-55)  discusses  database  vendor  statistics  field-testing; 

• Part  4 (pp.  56-80)  contains  the  data  collection  manual.  Appendix  C (pp.  96- 
101)  has  sample  forms. 

□ If  you  are  primarily  interested  in  the  manual,  we  encourage  you  to  read  the 
Project  Overview  to  understand  the  contexts  and  development  of  the  statistics 
and  measures  before  you  read  the  manual.  The  manual  includes  a definition, 
rationale,  procedures,  and  related  issues  concerning  collecting  and  using  each 
statistic  and  performance  measure. 

□ For  a detailed  discussion  of  the  issues  related  to  the  need  for  and  use  of 
statistics  and  measures  of  networked  services  in  academic  and  research 
libraries,  please  review  part  1.  It  provides  an  introduction  to  the  E-Metrics 
project  as  well  as  other  related  initiatives.  For  a discussion  of  the  outcomes- 
related  evaluation  of  library  services,  please  review  section  1.4. 

□ If  you  are  interested  in  the  field-testing  of  proposed  statistics  and  performance 
measures  at  the  participating  libraries,  please  review  part  2,  which  includes 
methodology,  results  and  lessons  learned. 

□ If  you  are  interested  in  the  database  vendor  statistics  portion  of  the  field- 
testing,  please  review  part  3. 

This  document  will  be  available  in  electronic  form  on  the  ARL  website  at 

http://www.arl.org/stats/newmeas/emetrics.  The  manual  (part  4)  and  training 

materials  will  be  available  in  January  2002  in  both  print  and  electronic  formats 

through  ARL. 
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PROJECT  OVERVIEW 

This  study  provides  one  approach,  a beginning  approach,  for  describing  and  measuring  some  of  the 
resources,  uses,  and  expenditures  for  supporting  networked  services  in  a research  library  setting.  Such 
statistics  and  measures  are  essential  for  collections  decisions;  cost  analysis;  justification  of  services; 
services  planning  and  evaluation;  and  a host  of  other  activities.  This  manual  is  a first  effort  to  accomplish 
these  objectives  and  to  standardize  data  collection  techniques,  definitions,  and  procedures  related  to 
network  and  electronic  resources  and  services. 

The  working  definition  of  networked  services  is  those  electronic  information  resources  and/or 
services  that  users  access  electronically  via  a computing  network  (I)  from  on-site  in  the  library  (2) 
remote  to  the  library,  but  from  a campus  facility,  or  (3)  remote  from  the  library  and  campus.  Examples 
of  networked  resources  include  local,  regional,  and  statewide  library  hosted  or  authored  web  sites  and 
library-licensed  databases  (e.g.,  InfoTrac,  EBSCOHost,  JSTOR,  Project  Muse). 

Examples  of  networked  services  include: 

• Text  and  numerical  databases,  electronic  journals  and  books; 

• Email,  listservs,  online  reference/assistance; 

• Training  in  the  use  of  these  resources  and  services; 

• Request  for  services  via  online  forms  (i.e.,  interlibrary  loans). 

The  range  and  types  of  services  accessible  through  and  supported  by  networks  will  continue  to  evolve  as 
network  technology  changes.  While  there  is  excitement  with  all  the  developments  related  to  the  provision 
of  networked  services,  there  are  a number  of  challenges  that  require  resolution  in  the  area  of  statistics  and 
measures  for  networked  services. 

This  manual  is  but  one  product  of  a larger  effort  to  develop  new  measures  and  evaluation  techniques 
by  the  Association  of  Research  Libraries  (ARL)  http://www.arl.ora/stats/newmeas/newmeas.html.  The 
New  Measures  Initiative  has  a number  of  projects  of  which  this  one,  The  E-Metrics  Project:  Developing 
Statistics  And  Performance  Measures  To  Describe  Electronic  Information  Services  And  Resources  For 
ARL  Libraries,  specifically  concentrates  on  developing,  field-testing,  and  refining  selected  core  statistics 
and  measures  to  describe  use  and  users  of  networked  services 

http://www.arl.org/stats/newmeas/emetrics/index.html.  This  manual  is  one  of  the  primary  products  from 
the  E-Metrics  project. 

Readers  of  this  manual  should  also  review  the  other  ARL  project  initiatives  in  this  area  (see 
http://www.arl.org/stats/newmeas/newmeas.html)  as  the  manual  is  best  seen  as  part  of  the  overall  work  in 
this  area.  Moreover,  use  of  this  manual  may  be  more  effective  when  coordinated  and  administered  in 
conjunction  with  other  ARL  New  Measures  initiatives. 
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E- Metrics  Project 

The  E-Metrics  project,  which  began  in  April  2000,  and  is  scheduled  for  completion  in  December  of 
2001,  is  funded  by  a group  of  24  ARL  member  libraries.  The  primary  goals  of  this  project  are  to: 

• Develop,  test,  and  refine  selected  statistics  and  performance  measures  to  describe  electronic 
services  and  resources  in  ARL  libraries; 

• Engage  in  a collaborative  effort  with  selected  database  vendors  to  establish  an  ongoing  means  to 
produce  selected  descriptive  statistics  on  database  use,  users,  and  services; 

• Develop  a model  to  describe  possible  relationships  between  library  activities  and 
library/institutional  outcomes; 

• Develop  a proposal  for  external  funding  to  maintain  the  development  and  refinement  of 
networked  statistics  and  performance  measures. 

This  report  addresses  the  first  three  of  these  goals.  An  interim  Phase  I Report  describing  current 
practices  of  participating  ARL  member  libraries  related  to  network  statistics  and  measures  was 
issued  November  7,  2000,  and  can  be  found  at: 
http://www.arl.org/stats/newmeas/emetrics/index.html. 

Other  key  components  of  the  E-Metrics  project  are  still  in  process.  The  study  team  is  developing  a 
model  to  describe  institutional  outcomes  and  the  degree  to  which  library  activities  may  be  related  to 
institutional  outcomes.  A final  product  from  the  E-Metrics  project  will  be  a proposal  that  may  be 
submitted  to  an  appropriate  funding  agency  to  continue  research  and  development  work  in  the  area  of 
statistics,  measurement,  and  research  library  outcomes.  That  proposal  will  be  completed  by  the  end  of 
December  2001.  Additional  information  regarding  the  project  goals,  objectives,  and  deliverables  is 
contained  in  the  statement  of  work  located  in  Appendix  A. 

Goals  and  Objectives  of  the  Manual 

As  with  all  projects,  the  manual  offered  here  has  a number  of  specific  goals  and  objectives.  Its 
primary  goal  is  to  provide  a beginning  approach  for  research  libraries  to  better  describe  the  use  and  users 
of  their  networked  services.  A secondary  goal  of  the  manual  is  to  increase  the  visibility  and  importance 
of  developing  such  statistics  and  measures.  Specific  objectives  of  the  manual  are  to: 

• Identify  selected  key  statistics  and  measures  that  can  describe  use  and  users  of  electronic  and 
networked  services; 

• Standardize  procedures  and  definitions  to  collect  these  statistics  and  measures; 

• Increase  awareness  of  selected  issues  related  to  collecting,  analyzing,  and  reporting  the  data  to 
produce  these  statistics  and  measures. 

The  study  team  expects  that  this  manual  will  be  a continually  evolving  product.  The  statistics  and 
measures  offered  here  will  continue  to  need  to  be  developed,  expanded,  refined,  and  possibly 
eliminated  over  time. 

This  manual  continues  and  extends  the  work  being  done  from  a number  of  previous  projects  (see 
discussion  in  Part  1).  The  manual  does  not , however: 
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• Offer  a comprehensive  list  of  statistics  and  measures  for  networked  services;  clearly  additional 
statistics  and  measures  are  possible  and  may  need  to  be  developed,  tested,  and  refined; 

• Make  specific  linkages  between  the  statistics  and  measures  described  in  this  manual  to  library 
and  institutional  outcomes; 

• Resolve  a number  of  definitional  and  procedural  issues  among  database  vendors  and  other 
standards  organizations  on  how  to  report  database  data  - continued  discussions  and  work  in  this 
area  are  needed; 

• Assist  library  decision-makers  in  determining  which  statistics  and  measures,  strategically  and 
politically,  would  be  best  to  use  in  a particular  library  setting; 

• Suggest  possible  organizational  structures  and  resources  needed  for  a library  to  successfully 
collect,  manage,  and  report  the  data. 

There  is  a range  of  situational  factors  and  data  needs/expectations  that  vary  considerably  from 
research  library  to  research  library.  This  manual  will  not  meet  all  those  needs  and  expectations.  Readers 
should  consider  this  manual  as  one  tool  (of  many)  that  can  assist  a library  in  describing  and  evaluating 
networked  services  use  and  users. 

Recommended  Statistics  And  Performance  Measures 

Based  on  a substantial  field-testing  process  (described  in  detail  in  this  report),  the  project  team 
recommends  the  following  network  statistics  (Table  P.l)  and  performance  measures  (Table  P.2).  The 
statistics  and  performance  measures  provide  indicators  of  library  networked  services  and  resources. 

Table  P.l  Recommended  Statistics 


Patron  Accessible  Electronic  Resources 

R1  Number  of  electronic  full-text  journals 
R2  Number  of  electronic  reference  sources 
R3  Number  of  electronic  books 

Use  of  Networked  Resources  and  Services 

U1  Number  of  electronic  reference  transactions 
U2  Number  of  logins  (sessions)  to  electronic  databases 
U3  Number  of  queries  (searches)  in  electronic  databases 
U4  Items  requested  in  electronic  databases 
U5  Virtual  visits  to  library's  website  and  catalog 

Expenditures  for  Networked  Resources 
and  Related  Infrastructure 

C 1 Cost  of  electronic  full-text  journals 
C2  Cost  of  electronic  reference  sources 
C3  Cost  of  electronic  books 

C4  Library  expenditures  for  bibliographic  utilities,  networks, 
and  consortia 

C5  External  expenditures  for  bibliographic  utilities,  networks, 
and  consortia 

Library  Digitization  Activities 

D 1 Size  of  library  digital  collection 
D2  Use  of  library  digital  collection 
D3  Cost  of  digital  collection  construction  and  management 

The  performance  measures  are  composite  and/or  combinations  of  the  above  network  statistics  along 
with,  in  some  cases,  non-network  statistics  already  collected  by  ARL  libraries  (e.g.,  number  of  visitors  to 
the  library). 
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Table  P.2  Recommended  Performance  Measures 


Performance  Measures 


PI  Percentage  of  electronic  reference  transactions  of  total  reference 
P2  Percentage  of  remote  library  visits  of  all  library  visits 
P3  Percentage  of  electronic  books  to  all  monographs 


Using  the  Network  Statistics  and  Performance  Measures 

The  recommended  network  statistics  and  performance  measures,  either  independently  or  in  some 
combination,  can  assist  research  libraries  in  describing  a number  of  aspects  of  their  networked  resources 
and  services.  This  section  provides  libraries  with  some  guidance  regarding  the  use  to  which  the  network 
statistics  and  measures  can  be  put. 

Although  the  statistics  and  measures  presented  in  this  manual  fall  under  a number  of  network 
components,  it  is  possible  to  categorize  broadly  the  statistics  and  measures  into  statistics  and  measures 
that  identify: 

1 . The  overall  size/volume  of  available  networked  resources.  By  collecting  and  reporting  the 
recommended  statistics,  libraries  are  able  to  identify  their  total  number  of  journals  available 
electronically  in  full-text  format  (e-joumals);  reference  sources  available  electronically  to  staff 
and  patrons;  books  available  electronically  (e-books);  and  items  digitized  from  the  library’s  own 
collection  (e.g.,  documents  digitized  and  mounted  through  a network  accessible  method).  Also, 
by  combining  R3  (number  of  electronic  books)  with  a count  of  library  print  monographs,  libraries 
can  determine  the  overall  percentage  of  books  available  in  electronic  format  to  the  populations 
that  they  serve  (P3). 

2.  The  extent  to  which  the  networked  resources  and  services  are  used  by  the  library ' s service 
population.  The  use  statistics  enable  libraries  to  identify  the  overall  number  of  sessions  to  library 
database  subscription  services,  with  a number  of  sub-categories  if  desired  (e.g.,  location,  per  title, 
etc.);  number  of  queries  conducted  by  users  of  the  database  subscription  services,  with  a number 
of  sub-categories  if  desired  (e.g.,  location,  per  title,  etc.);  number  of  items  requested  (e.g.,  printed, 
e-mailed,  saved,  or  otherwise  accessed)  by  users  of  the  database  subscription  services,  with  a 
number  of  sub-categories  if  desired  (e.g.,  location,  per  title,  etc.);  number  of  visits  to  library- 
maintained  virtual  resources  (e.g.,  web  pages),  with  a number  of  sub-categories  if  desired  (e.g., 
location  - virtual  v.  in-library);  number  of  accesses  and  queries  conducted  by  library  service  users 
of  library-maintained  virtual  resources,  with  a number  of  sub-categories  if  desired  (e.g.,  location 
- virtual  v.  in-library);  and  number  of  electronic  reference  transactions  conducted. 

3.  Through  a combination  of  the  U 1 (number  of  electronic  reference  transactions)  and  U5  (virtual 
visits)  network  statistics  and  traditional  statistics  of  reference  transactions  and  library  visitors, 
libraries  can  gain  a sense  of  the  ratio  of  electronic  reference  to  total  reference  transactions,  as  well 
as  the  ratio  of  virtual  (remote)  library  visits  to  physical  library  visits.  These  provide  important 
trend  indicators  for  the  use  of  selected  networked  library  resources  and  services. 

4.  The  cost  to  the  library  of  providing  the  networked  resources  to  its  service  population.  The 
statistics  provide  libraries  with  cost  data  that  indicate  the  cost  of  subscribing  to  and/or  purchasing 
online  full-text  journals;  the  cost  of  subscribing  to  and/or  purchasing  online  reference  sources;  the 
cost  of  subscribing  to  electronic  books  (e-books);  library  expenditures  for  online  material 
provided  through  participation  in  local,  regional,  and/or  national  consortia  NOT  included  in 
database  subscription  services;  the  expenditures  of  non-library  entities  (e.g.,  state-wide  resources 
provided  through  consortia  or  governmental  agencies)  on  behalf  of  libraries  for  subscription 
services;  and  the  costs  associated  with  library  digitization  activities — including  equipment, 
software,  contracted  services,  personnel,  other). 
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The  use  of  these  statistics,  particularly  in  combination,  can  enable  research  libraries  to  answer  a 
number  of  questions  regarding  network  services  and  resources  as  detailed  in  the  manual. 

A Note  on  the  Vendor  Statistics 

The  vendor  field-test  component  of  the  project  identified  a number  of  process,  administrative,  and 
management  issues  regarding  the  collection,  manipulation,  and  reporting  of  vendor-supplied  online 
database  usage  data: 

• Are  the  data  provided  to  libraries  reliable? 

Since  the  field-testing  dealt  with  only  one  month’s  worth  of  data,  it  is  difficult  to  answer  the 
question.  On  the  other  hand,  we  have  not  heard  from  field-testing  libraries  of  any  usual 
discrepancy  between  the  field-testing  data  and  data  they  had  received  before  the  field-testing. 

The  study  team  realizes  that  simply  comparing  data  from  the  same  vendors  will  not  give  us  a 
satisfactory  answer.  During  the  course  of  writing  this  report,  we  came  across  an  email  message 
from  a major  database  vendor  acknowledging  errors  in  their  usage  reports.  While  the  vendor  did 
the  right  thing  by  admitting  their  fault,  it  shows  us  that  libraries  are  not  in  a good  position  to 
know  about  what  goes  into  the  vendor  reports.  Some  unusual  numbers  or  patterns  are  relatively 
easy  to  identify.  But  consistent  under  - or  over  - counts  will  be  hard  to  detect. 

One  way  to  deal  with  the  reliability  issue  is  for  libraries  to  collect  data  in-house.  For  example, 
some  libraries  have  set  up  redirect  webpages  for  external  databases  to  count  the  number  of 
attempted  logins  to  licensed  databases.  This  kind  of  data  can  be  used  to  cross-check  vendor- 
supplied  numbers.  Also,  the  library  community  needs  to  consider  concrete  ways  (e.g.,  third  party 
validation)  to  ensure  consistent  and  reliable  reporting  from  the  vendors,  or  at  least  should  demand 
better  documentation  of  the  data  collection  and  filtering  process  from  the  vendors. 

• Are  the  data  comparable  across  libraries,  products,  and  vendors? 

Use  of  different  system  parameters  (e.g.,  time-out),  the  application  of  different  assumptions  on 
user  behaviors  (e.g.,  how  to  treat  or  count  multiple  clicks  on  the  same  document  within  a 
session),  and  the  lack  of  adequate  explanation  in  vendor  documentation  regarding  specific 
definitions  and  data  collection  and  filtering  processes  all  contribute  to  the  problem.  Therefore,  it 
is  largely  impossible  to  compare  data  across  vendors  and  as  a result,  comparison  should  be 
limited  to  data  from  the  same  vendors.  The  comprehensive  standardization  of  usage  statistics  and 
data  delivery  methods  (e.g.,  file  format  and  data  arrangement)  cannot  be  easily  achieved  in  the 
short-term.  Those  are  the  long-term  goals  for  which  vendors  and  libraries  need  to  work  together. 
The  ARL  community  should  continue  to  make  progress  in  this  area  by  working  amongst 
themselves  and  with  the  database  vendor  community. 

• Are  the  data  easy  to  obtain  and  manipulate? 

We  believe  that  the  data  provided  by  the  vendors  studied  are  easy  to  obtain  and  manipulate.  Most 
vendors  offer  several  data  formats  including  text  format  (e.g.,  comma  separated  file)  and 
spreadsheet  format  (e.g.,  MS  Excel)  in  addition  to  standard  HTML  format  for  easy  viewing  in 
web  browsers.  Also,  many  vendors  offer  ad-hoc  report  generation  whereby  libraries  can 
customize  the  fields  they  want  to  get  and  set  desired  time  periods. 
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However,  processing  vendor  reports  from  multiple  vendors  can  be  a considerable  burden  on 
libraries,  in  terms  of  time  and  staff  efforts,  as  the  formats  and  data  arrangements  vary 
considerably  from  vendor  to  vendor.  Therefore,  vendors  should  report  standardized  usage 
statistics,  such  as  the  ones  recommended  by  the  ICOLC,  in  one  report  in  the  standardized  column 
and  row  arrangements  and  provide  a separate  report  that  contains  vendor  specific  additional  data. 

• Do  the  data  provide  meaningful  information  about  the  usage  of  networked  information  resources? 

Usage  statistics  currently  being  provided  by  vendors  give  useful  information  regarding  the 
utilization  of  external  subscription-basis  information  services.  Libraries  use  data  for  a variety  of 
purposes:  usage  trends  over  time,  justification  for  expenditures,  cost  analysis,  modification  of 
service  provision.  Related  to  the  issue  of  the  value  of  data  is  trustworthiness  (reliability)  of  data. 
Also,  there  is  some  concern  over  the  lack  of  user-related  information  in  usage  statistics. 

These  issues  point  to  a number  of  recommendations  included  in  this  report.  They  also  reiterate 
the  importance  of  continued  ARL  collaboration  with  the  vendors  and  the  library  community  as  a 
whole  (e.g.,  ICOLC,  NISO,  ISO,  publishers)  as  there  is  substantial  interest  in  online  database 
usage  statistics. 


Institutional  Outcomes 

University  research  libraries  are  established  to  support  the  broad  research,  education,  and  service 
goals  that  are  fundamental  to  the  mission  of  the  institutions  they  serve.  Beyond  helping  to  fulfill  the 
university  mission,  a research  library  must  be  able  to  help  the  larger  institution  reach  its  more  concrete 
but  shifting  goals.  These  goals  may  be  articulated  in  strategic  planning  documents,  in  conversations  with 
the  provost  or  academic  deans,  as  part  of  regional  accreditation  standards,  or  in  state  legislation. 

It  is  important  for  the  library  to  be  aware  of  these  goals  and  to  be  able  to  target  library  resources, 
services,  and  programs  to  help  meet  institutional  goals.  Doing  so  is  critical  to  the  library  being  perceived 
as  a vital,  contributing  part  of  the  university.  To  address  these  goals  and  measure  them  effectively,  it  is 
important  to  ask  three  key  questions: 

1 . What  is  the  desired  state  of  the  university? 

2.  How  can  the  library  help  the  university  to  achieve  this  state? 

3.  How  will  the  library  know  when  it  has  been  successful  in  helping  the  university  achieve  this 
state? 

The  first  question  helps  the  library  better  understand  the  operating  environment.  Knowing  what  the 
university  wants  to  be  now  and  in  the  future  helps  library  administrators  understand  what  customers  need. 

The  second  question  helps  the  library  in  making  decisions  about  which  of  the  many  goals  of  the 
university  are  helpful  for  the  library  to  focus  on.  Also,  answering  this  question  helps  the  library  in 
making  decisions  about  what  action  it  must  take  to  contribute  to  the  fulfillment  of  those  goals.  Existing 
services  and  programs  may  address  them,  programs  may  have  to  be  fine-tuned  and  resources  upgraded,  or 
new  programs  and  resources  may  need  to  be  developed  and  acquired. 

The  third  question  helps  the  library  craft  the  measures  that  will  provide  indications  of  success.  This  is 
extremely  important  and  must  be  considered  carefully  because  direct  outcomes  measurement  is  often 
difficult  or  impossible.  It  may  be  necessary  to  develop  several  measures  that  work  together  to  indicate  an 
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outcome  or  to  use  surrogate  measures,  such  as  perception  surveys.  One  framework  for  developing  such 
measures  is  described  below. 

The  order  in  which  these  questions  are  addressed  is  also  vital.  Before  measurements  can  be  derived, 
library  administrators  must  know  what  impact  their  library  can  have  on  matters  of  importance  to  the 
university  and  to  the  library.  Without  this  perspective,  outcomes  measurement  loses  meaningfulness.  In 
other  words,  it  is  very  difficult  to  gain  useful  insights  about  outcomes  when  measures  are  not  designed 
with  outcomes  explicitly  in  mind. 

ARL  libraries  may  currently  collect  measures  that  can  provide  some  indication  of  the  success  of  a 
particular  program  or  service  provided  to  customers,  such  as  user  satisfaction  surveys;  however,  it  is 
important  to  think  broadly — with  the  desired  state  in  mind — and  not  simply  use  the  measures  at  hand 
because  they  are  easy  to  collect  or  because  a lot  of  time  and  effort  has  been  devoted  to  collecting  them. 

In  most  cases,  a single  measure  on  its  own  is  not  enough  to  indicate  whether  a research  library  is 
successful  in  a given  area.  To  accurately  indicate  the  success  or  quality  of  an  academic  library, 
measurement  should  be  implemented  at  three  key  levels  (see  Figure  ES-2): 

• Outcome  Level 

• Use/Capacity  Level  (Output) 

• Resource  level  (Input) 

Following  this  approach,  however,  may  lead  to  the  formulation  of  a wide  range  of  performance 
measures  and  statistics.  Selection  of  the  precise  measures  needed  to  evaluate  an  electronic 
resource  or  service  can  be  especially  difficult,  even  for  libraries  that  have  undertaken  processes 
similar  to  those  described  above.  Therefore,  it  is  important  to  have  a framework  to  assist  in 
choosing  measures  to  gain  insights  into  the  use  and  users,  management,  and  reach  of  networked 
services  and  resources  in  specific  areas  or  across  a number  of  areas. 


Figure  P.  1 Using  Measures  to  Answer  Questions  at  Different  Levels 


Outcome  Level 

Use/Capacity  Level 
(Output  Measures) 

Resource  Level 
(Input  Measures) 

What  are  the  results  of  a 
program  or  process? 

How  much  is  a service,  resource,  or 
program  being  used? 

What  do  we  need  to  ensure 
success? 

How  successful  or  effective  is 
the  library? 

Who  is  using  a service,  resource,  or 
program? 

What  funding  level  is 
appropriate  or  necessary  for  a 
particular  program? 

How  effective  do  customers 
perceive  your  programs  to  be? 

Why  are  people  using  a particular 
program? 

Do  we  need  more  of  a 
particular  resource  in  order  to 
have  a more  effective 
program? 

What  beneficial  effects  are  you 
having  on  your  customers? 

How  could  a program  be 
changed  to  better  suit  the  needs 
of  your  customers? 

The  Importance  of  Developing  Outcomes  Processes 

As  discussed  in  the  preceding  section,  ARL  libraries  need  to  develop  a process  to  identify  and 
operationalize  library  outcomes  that  contribute  to  institutional  outcomes.  The  library  plays  a critical  role 
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in  informing  the  university  of  valued  institutional  outcomes  to  which  the  library  contributes.  Setting  up 
such  a process  allows  the  library  to  inform  key  stakeholders  in  the  university  of  both  the  library’s  role  in 
institutional  outcomes  and  insures  that  the  institutional  outcomes  to  which  the  library  has  (or  may  have) 
links  are  in  fact  appropriate. 

Because  each  university  has  different  processes  for  information  sharing,  decision-making,  and 
mission  fulfillment,  it  is  important  that  each  university  library  identify,  understand,  and  master  the 
established  local  process.  It  is  a given  that  libraries  must  work  within  their  particular  organizational 
framework.  Therefore,  to  maximize  contributions  to  university  outcomes,  the  library  must  orient  itself  to 
and  operate  with  the  fullest  advantages  of  that  local  framework. 

An  important  factor  that  contributes  to  an  effective  understanding  of  the  local  situation  is  sensitivity 
to  the  differing  viewpoints  of  various  stakeholder  groups.  What  might  constitute  institutional  outcomes, 
or  appropriate  institutional  outcomes,  will  vary  from  one  group  to  another  (deans,  faculty,  trustees, 
students,  etc.).  It  may  be  that  the  outcomes  from  the  library  (as  agreed-upon  by  library  staff  and 
administration)  may  or  may  not  be  those  seen  by  university  administration  as  important  or  appropriate. 
Developing  a process  to  address  these  potentially  conflicting  stakeholder  concerns  at  the  local  level  is 
particularly  crucial. 


Key  Issues  In  The  Networked  Environment 

In  general,  despite  the  fact  that  many  of  the  recommended  statistics  are  gross  figures  and  concerned 
mostly  with  resource  counts  and  costs,  data  collection  is  not  an  easy  process.  There  are  a number  of 
issues  and  challenges  that  affect  the  library’s  ability  to  collect  statistics  and  measures  to  describe  its 
electronic  resources  and  services: 

• Acquisitions,  accounting,  and  cataloging  systems  are  not  set  up  to  support  this  kind  of  data 
collection.  Current  bibliographic  and  management  information  systems,  for  the  most  part,  reflect 
practices  in  the  pre-Web,  print-dominant  environment.  It  appears  that  providing  access  to 
electronic  resources  is  keeping  many  research  libraries  busy  enough  already.  The  lack  of 
efficient  information  systems  that  pull  together  elementary  data  elements  forced  many  field- 
testing  libraries  to  resort  to  labor-intensive  processes  to  collect  data.  According  to  a recent 
survey  done  by  Tim  Jewell  at  the  University  of  Washington  Libraries,  there  are  about  10  ARL 
libraries  that  have  a production  system  for  managing  electronic  resources,  and  several  others  in 
the  planning  or  development  stage  (http://www.library.comell.edu/cts/elicensestudy/home.html). 
While  these  systems  are  not  developed  solely  for  data  collection  purposes,  they  certainly  facilitate 
data  collection  efforts  such  as  the  E-Metrics  project.  In  the  absence  of  such  fully  developed 
information  systems,  we  advise  ARL  libraries  to  develop,  at  a minimum,  an  in-house  spreadsheet 
or  database  file  to  keep  track  of  key  data  elements  related  to  electronic  resources  and  services. 

• Prescribed  definitions  and  procedures  are  not  compatible  with  local  practices . Several  field- 
testing  libraries  independently  have  been  collecting  some  of  the  similar  statistics  and  measures, 
but  their  definitions  and  promulgation  of  the  methodologies  differ  from  what  the  field-testing 
entailed.  It  seems  that  the  majority  of  libraries  want  to  build  their  local  procedures  in  sync  with 
the  standardized  ARL  practices,  a sentiment  that  is  echoed  in  the  following  comment:  “We  will 
adjust  our  in-house  practices  to  be  able  to  report  in  this  way.”  The  data  collection  manual 
produced  from  this  study  is  one  step  in  that  direction. 

• The  nature  of  electronic  resources  and  services  is  still  fluid  and  makes  it  difficult  to  devise  clear- 
cut  definitions  and  procedures.  For  instance,  as  several  people  have  already  argued  (Snowhill, 
2001;  Sottong,  2001),  the  concept  of  electronic  books  is  still  evolving  due  to  changes  in 
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technology,  the  market,  and  use  of  resources  among  other  things.  As  an  illustration,  think  of  the 
full-text  search  capability  in  most  electronic  books.  It  can  be  argued  that  there  is  no  clear 
distinction  between  electronic  books  and  reference  sources,  especially  from  the  user’s  point  of 
view.  Electronic  access  can  trigger  an  entirely  new  conceptualization  of  a given  information 
object  as  in  the  case  of  electronic  books.  Libraries  need  to  deal  with  the  implications  of  this 
changing  environment  and  be  more  flexible.  We  acknowledge  that  the  distinction  made  for 
different  electronic  resources  in  the  study  and  in  the  current  E-Metrics  work  is  only  temporary 
and  will  have  to  be  revised  as  we  progress. 

• The  dispersed  nature  of  resources  in  the  networked  environment  makes  it  difficult  to  consolidate 
and  manage  statistics.  It  is  also  a growing  source  of  frustration  for  many  librarians  who  deal 
with  electronic  resources.  Various  listservs  devoted  to  electronic  resources  and  voluminous 
correspondence  in  the  listservs  reflect  this  trend.  Traditionally,  library  materials,  with  a notable 
exception  of  government  publications,  are  centrally  managed  through  a library  catalog.  Also, 
library  visit  counts  have  traditionally  been  normalized  by  using  turnstile  counts  whenever 
possible.  However,  in  the  networked  environment,  libraries  have  to  deal  with  a whole  range  of 
resources  and  access  points.  This  in  turn  creates  more  complexity  in  not  only  managing 
resources  but  also  collecting  data  about  the  resources  and  their  use.  For  example,  with  respect  to 
usage  statistics  of  licensed  materials,  while  setting  up  a library  database  gateway  may  allow  the 
library  to  collect  a coherent  statistic  (e.g.,  attempted  logins  to  licensed  databases),  it  does  not 
account  for  traffic  that  goes  directly  to  vendor  websites.  On  the  other  hand,  usage  statistics  from 
database  vendors  are  more  complete  in  the  sense  that  they  capture  all  requested  use  of  the 
database,  but  the  incompatibility  of  statistics  from  various  vendors  makes  it  difficult  for  the 
libraries  to  compare  and  aggregate  usage  data.  Therefore,  it  is  important  that  libraries  be  able  to 
deal  with  incomplete,  incompatible  data  from  multiple  sources  and  make  the  best  decisions  based 
on  the  given  data. 

• There  are  a number  of  definitional  and  procedural  issues  among  database  vendors , library 
consortia  (e.g,  ICOLC),  and  other  standards  organizations  (e.g.,  NISO,  ISO)  on  how  to  report 
database  usage  statistics.  Working  with  major  database  vendors  is  one  of  the  important  areas  to 
concentrate  on  in  the  future.  Our  study  initiated  dialogue  with  selected  vendors  and  their 
involvement  proved  to  be  very  useful  and  needs  to  be  continued. 

• The  findings  indicate  that  there  are  varying  levels  of  resources  and  support  available  in  the 
libraries  to  support  data  collection  and  reporting.  The  degree  to  which  libraries  will  be  able  to 
collect  these  data  and  use  them  is  linked  to  the  amount  of  resources  they  can  commit. 

• There  are  a range  of  situational  factors  and  data  needs/expectations  that  vary  considerably  from 
research  library  to  research  library.  Individual  libraries  will  need  to  determine  which  statistics 
and  measures  would  be  best  to  use,  strategically  and  politically,  in  their  own  settings.  They  will 
also  need  to  consider  possible  organizational  structures  and  resources  needed  to  successfully 
collect,  manage,  and  report  the  data. 

These  issues  are  not  insurmountable.  They  require,  however,  additional  research  efforts  as  well 
as  continued  effort  by  libraries  to  collect,  analyze,  and  use  the  network  statistics  and  performance 
measures  for  management  and  decision-making  purposes. 


Expanding  Measurement  Tools 

The  explosion  of  networked  information  services  has  been  relatively  recent  and  the  impact  from  this 
increase  of  services  and  the  corresponding  technology  is  only  beginning  to  be  understood.  An  ever- 
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increasing  portion  of  library  collections’  dollars  is  committed  to  purchasing  networked  services.  Yet 
relatively  little  is  known  about  how  these  services  are  used,  who  uses  them,  and  what  impact  these 
services  have. 

Many  research  libraries  possess  inadequate  resources,  staffing,  and  expertise  to  collect,  manage,  and 
report  the  data  related  to  describing  networked  services.  For  these  libraries,  some  organizational 
development  and  commitment  to  collecting  and  using  these  data  may  be  necessary  to  take  advantage  of 
the  measurement  tools  and  techniques  outlined  in  part  4.  Nonetheless,  the  discussion  of  the  measurement 
issues  in  parts  1-3  can  assist  these  libraries  to  better  understand  why  such  measurement  is  essential. 

Given  the  rapidly  changing  technology  environment,  the  changing  milieu  affecting  higher  education, 
changing  organizational  structures  within  ARL  libraries,  and  the  complexity  of  measuring  such 
networked  services,  it  is  almost  certain  that  the  statistics  and  measures  proposed  in  this  study  will 
continue  to  evolve.  The  measurement  tools  offered  in  this  report,  however,  will  provide  research 
librarians  with  important  techniques  to  count,  describe,  and  report  networked  services  and  resources  in 
their  libraries. 
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1 CONTEXT  AND  ISSUES 

This  report  is  but  one  product  of  a larger  effort  by  the  Association  of  Research  Libraries  (ARL)  to 
develop  new  measures  and  evaluation  techniques.  The  New  Measures  Initiative  includes  a number  of 
projects,  of  which  this  one,  The  E-Metrics  Project:  Developing  Statistics  And  Performance  Measures  To 
Describe  Electronic  Information  Services  And  Resources  For  ARL  Libraries,  specifically  concentrates  on 
developing,  field-testing,  and  refining  selected  core  statistics  and  measures  to  describe  use  and  users  of 
networked  services  (http://www.arl.org/stats/newmeas/emetrics/index.htmn.  This  manual  is  one  of  the 
primary  products  from  the  E-Metrics  project.  As  such,  it  represents  a first  effort  to  accomplish  these 
objectives  and  to  standardize  data  collection  techniques,  definitions,  and  procedures  related  to  network 
and  electronic  resources  and  services.  Readers  of  this  manual  should  also  review  the  other  ARL  project 
initiatives  in  this  area  (see  http://www.arl.org/stats/newmeas/newmeas.htmn  as  the  manual  is  best  seen  as 
part  of  the  overall  work  in  this  area.  Moreover,  use  of  this  manual  may  be  more  effective  when 
coordinated  and  administered  in  conjunction  with  other  ARL  New  Measures  initiatives. 

The  E-Metrics  project  began  in  April  2000,  is  scheduled  for  completion  in  December  of  2001,  and  is 
funded  by  a group  of  24  ARL  member  libraries.  The  primary  goals  of  this  project  are  to: 

• Develop,  test,  and  refine  selected  statistics  and  performance  measures  to  describe  electronic 
services  and  resources  in  ARL  libraries; 

• Engage  in  a collaborative  effort  with  selected  database  vendors  to  establish  an  ongoing  means  to 
produce  selected  descriptive  statistics  on  database  use,  users,  and  services; 

• Develop  a model  to  describe  possible  relationships  between  library  activities  and 
library/institutional  outcomes;  and 

• Develop  a proposal  for  external  funding  to  maintain  the  development  and  refinement  of 
networked  statistics  and  performance  measures 

This  report  addresses  the  first  three  of  these  goals.  An  interim,  Phase  I Report  describing  current 
practices  of  participating  ARL  member  libraries  related  to  network  statistics  and  measures  was 
issued  November  7,  2000,  and  can  be  found  at: 

http://www.arl.org/stats/newmeas/emetrics/index.html.  Appendix  A contains  the  original  study 
proposal. 

Other  key  components  of  the  E-Metrics  project  are  still  in  process.  The  study  team  developed  a 
model  to  describe  institutional  outcomes  and  the  degree  to  which  library  activities  may  be  related  to 
institutional  outcomes.  A final  product  from  the  E-Metrics  project  will  be  a proposal  that  may  be 
submitted  to  an  appropriate  funding  agency  to  continue  research  and  development  work  in  the  area  of 
statistics,  measurement,  and  academic  library  outcomes.  That  proposal  will  be  completed  by  the  end  of 
December  2001. 
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1.1  Development  of  The  Manual 

The  manual  (Part  4 of  this  report)  is  based  on  a number  of  earlier  activities.  Initially,  the  study  team 
conducted  a number  of  site  visits  to  determine  best  practices  at  selected  ARL  libraries  regarding 
networked  services  statistics  and  measures.  This  activity  resulted  in  the  Phase  1 Report.  The  Phase  I 
Report  provided  an  important  context  for  the  current  use  of  statistics  and  measures  by  participating 
members.  Equally  important,  it  provided  a basis  for  the  study  team  to  propose  possible  statistics  and 
measures  that  would  best  meet  the  needs  of  participating  members. 

While  that  activity  progressed,  the  study  team  set  in  motion  a process  to  meet  with  selected  database 
vendors  to  begin  a dialogue  about  how  to  best  coordinate  and  standardize  data  collection  procedures  and 
definitions  so  that  libraries  could  compare  and  better  use  data  provided  by  these  vendors.  This  process, 
which  continued  through  2001,  involved  identifying  appropriate  individuals,  meeting  with  them,  and 
scheduling  group  meetings. 

In  addition,  the  discussions  about  these  data  collection  procedures  were  coordinated  with  other 
organizations  such  as  NISO  (National  Information  Standards  Organization),  ICOLC  (International 
Coalition  of  Library  Consortia),  and  NCLIS  (National  Commission  on  Libraries  and  Information 
Science).  Additional  input  came  from  the  study  team's  work  on  developing  a model  to  describe  the 
relationship  between  library  activities  and  larger  institutional  outcomes. 

Participating  members  in  the  E-Metrics  project  reviewed  proposed  statistics  and  measures.  They  then 
provided  the  study  team  with  feedback  as  to  which  would  be  most  useful  and  important.  This  information 
was  provided  through  a number  of  members  and  via  the  project  discussion  list.  A preliminary  list  of 
statistics  and  measures  were  field-tested  by  a number  of  members.  A member  of  the  study  team  visited 
selected  field-test  sites  to  better  understand  how  the  statistics  and  measures  were  being  used.  Throughout 
the  process  the  study  team  consulted  other  experts  in  the  field  as  well  as  appropriate  writings  and  research 
on  the  topic. 

Participating  members  have  had  significant  and  on  going  input  into  the  development  of  these  statistics 
and  measures.  Indeed,  were  it  not  for  this  involvement,  the  manual  could  not  have  been  produced.  Based 
on  these  and  other  activities,  the  study  team  developed  a draft  manual  that  was  then  reviewed  by  selected 
experts.  The  participation  by  members  and  the  reviewer  comments,  as  well  as  the  review  by  the  study 
team  itself,  led  to  the  version  of  the  manual  presented  here. 

1.1.1  Objectives  of  the  Manual 

As  with  all  projects,  the  manual  offered  in  part  4 has  a number  of  specific  goals  and  objectives.  Its 
primary  goal  is  to  provide  a beginning  approach  for  research  libraries  to  better  describe  the  use  and 
users  of  their  networked  services.  A secondary  goal  of  the  manual  is  to  increase  the  visibility  and 
importance  of  developing  such  statistics  and  measures.  Specific  objectives  of  the  manual  are  to: 

• Identify  selected  key  statistics  and  measures  that  can  describe  use  and  users  of  electronic  and 
networked  services; 

• Standardize  definitions  of  these  statistics  and  measures  and  standardize  procedures  for  collecting 
them; 

• Increase  awareness  of  selected  issues  related  to  collecting,  analyzing,  and  reporting  the  data  to 
produce  these  statistics  and  measures. 
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The  study  team  expects  that  this  manual  will  be  a dynamic  product.  The  statistics  and  measures 
offered  here  will  continue  to  need  to  be  developed,  expanded,  refined,  and  possibly  eliminated 
over  time. 

A number  of  objectives  are,  however,  beyond  the  scope  of  this  manual.  The  manual  does  not,  for 


• Offer  a comprehensive  list  of  statistics  and  measures  for  networked  services  - clearly  additional 
statistics  and  measures  are  possible  and  may  need  to  be  developed,  tested,  and  refined; 

• Address  measurement  of  service  quality; 

• Make  specific  linkages  between  the  statistics  and  measures  described  in  this  manual  to  library  and 
institutional  outcomes; 

• Resolve  a number  of  definitional  and  procedural  issues  among  database  vendors  and  other 
standards  organizations  on  how  to  report  database  data  - continued  discussions  and  work  in  this 
area  are  needed; 

• Assist  library  decision-makers  in  determining  which  statistics  and  measures,  strategically  and 
politically,  would  be  best  to  use  in  a particular  library  setting; 

• Suggest  possible  organizational  structures  and  resources  needed  for  a library  to  successfully 
collect,  manage,  and  report  the  data. 

Situational  factors  and  data  needs/expectations  vary  considerably  from  research  library  to  research 
library.  While  there  are  many  exciting  opportunities  related  to  the  provision  of  networked  services,  there 
are  a number  of  challenges  that  require  resolution  in  the  area  of  statistics  and  measures  for  networked 
services.  This  manual  provides  one  approach,  a beginning  approach,  to  offer  procedures  for  describing 
and  measuring  some  of  the  uses  and  users  of  networked  services  in  a research  library  setting. 


This  report  has  four  major  parts: 

• Part  1 provides  an  overall  introduction  to  the  importance  of  statistics  and  measures  in  a 
networked  services  environment,  a description  of  some  of  the  characteristics  of  that  environment, 
and  an  explanation  of  how  those  characteristics  may  impact  the  use  and  application  of  statistics 
and  measures. 

• Part  2 provides  a description  of  the  evolving  nature  of  statistics  in  a networked  library 
environment  and  a description  of  the  field-testing  methods  and  procedures  used  by  the  study 
team. 

• Part  3 describes  activities  that  have  been  accomplished  by  the  study  team  in  cooperation  with 
vendors  and  project  participants  related  to  usage  statistics  provided  by  vendors. 

• Part  4 is  a procedural  manual  for  the  collection  and  implementation  of  the  statistics  and  measures. 
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1.2  Importance  of  Statistics  and  Measures  for  Networked  Services 


1.2.1  need  for  Networked  Statistics 

The  development  of  library  networked  statistics  and  performance  measures  is  receiving 
increased  attention  and  support.  There  is  a broad  recognition  for  the  need  of  network  statistics 
and  performance  measures  that: 


Assist  libraries  in  making  a strong  case  for  support  of  technology  and  information  infrastructure 
by  documenting  their  Internet-based  services  and  resources; 

Assist  libraries  in  demonstrating  the  use  of  digital  collections  in  order  to  make  a case  for 
continued  collection  development  and  support; 

Allow  libraries  to  effectively  compare  themselves  to  others  in  terms  of  Internet-based  collection 
and  service  development,  costs,  provision  of  services,  connectivity,  and  use; 

Allow  libraries  to  measure  and  track  internal  changes  to  library  operations  as  well  as  uses  and 
users  of  library  resources  and  services; 

Enable  library  directors  and  administrative  library  agencies  to  compete  for  resources  with  other 
organizations  and/or  departments  by  documenting  the  range,  extent,  and  impact  of  library- 
provided  networked  services; 


Facilitate  the  expansion  from  traditional  library  use  measures  such  as  circulation,  reference 
transactions,  interlibrary  loans,  etc.,  to  include  network  measures  that  describe  the  nature  and  use 
of  library-based  network  activities  and  resources; 


Provide  a decision-making  framework  for  library  staff,  managers, 
resource  allocation  strategies  and  meet  other  management  needs; 


and  administrators  to  determine 


Provide  a means  through  which  to  measure  the  quality  of  library  services  and  resources  in  the 
networked  environment. 


These  and  other  factors  point  to  the  overall  importance  for  the  development,  collection,  and  reporting 
of  library  network  statistics  and  performance  measures  to  facilitate  collections  decisions,  cost  analysis, 
justification  of  services,  services  planning  and  evaluation,  and  a host  of  other  activities. 

The  working  definition  of  networked  services  is  those  electronic  information  resources  and/or 
services  that  users  access  electronically  via  a computing  network  (1)  from  on-site  in  the  library,  (2) 
remote  to  the  library,  but  from  a campus  facility,  or  (3)  remote  from  the  library  and  campus.  Examples 
of  networked  resources  include  local,  regional,  or  statewide  library  hosted  or  authored  web  sites  or 
library-licensed  databases  (e.g.,  InfoTrac,  EBSCOHost,  JSTOR,  Project  Muse).  Examples  of  networked 
services  include: 


• Text  and  numerical  databases,  electronic  journals  and  books; 

• Email,  listservs,  online  reference/assistance; 

• Training  in  the  use  of  these  resources  and  services;  and 

• Request  of  services  via  online  forms  (i.e.,  interlibrary  loans). 


The  range  and  types  of  services  accessible  through  and  supported  by  networks  will  continue  to  evolve 
as  network  technology  changes. 
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1.2.2  Current  Context  for  Networked  Services  Measurement 

An  earlier  effort  by  McClure  and  Lopata  (1996)  offered  a number  of  strategies  and  measures  to  assess 
the  academic  networked  environment.  But,  the  explosion  of  networked  information  services  has  been 
relatively  recent  and  the  impacts  from  this  increase  of  services  and  the  corresponding  technology  are  only 
beginning  to  be  understood.  An  ever-increasing  portion  of  library  collections  dollars  are  committed  to 
purchasing  networked  services.  Yet  relatively  little  is  known  about  how  these  services  are  used,  who  uses 
them,  and  what  the  overall  impact  of  these  services  is. 

Implementing  effective  measurement  standards  and  techniques  is  complicated  by  the  technology 
infrastructure  in  place  at  many  ARL  libraries,  the  membership  of  libraries  in  larger  consortia  that  share 
access  to  (and  cost  of)  networked  services,  and  the  various  configurations  used  by  vendors  to  supply  these 
networked  services.  In  addition,  as  found  in  the  Phase  I Report,  many  research  libraries  simply  have 
inadequate  resources,  staffing,  and  expertise  to  collect,  manage,  and  report  the  data  related  to  describing 
networked  services. 

As  often  stated  by  Carla  Stoffle,  Dean  of  Libraries  at  the  University  of  Arizona  and  current  Chair  of 
the  ARL  Statistics  and  Measurement  Committee,  “Developing  measures  and  evaluation  techniques  for 
networked  services  will  take  time,  effort,  and  on  going  learning  on  everyone’s  part  - but  we  must  begin 
now".  Given  the  rapidly  changing  technology  environment,  the  changing  milieu  affecting  higher 
education,  changing  organizational  structures  within  ARL  libraries,  and  the  complexity  of  measuring  such 
networked  services,  it  is  likely  that  the  statistics  and  measures  proposed  in  this  manual  will  continue  to 
evolve.  This  manual,  therefore,  serves  as  an  important  first  effort  in  establishing  standards  for  such 
statistics  and  measures. 

1.3  Conceptual  Framework 

This  section  of  the  report  (1)  provides  a framework  to  assist  libraries  to  select,  use,  and  understand 
network  statistics  and  performance  measures  based  on  various  aspects  of  a library’s  network(s);  (2) 
provides  a rationale  for  collecting  and  managing  network  statistics  and  measures;  and  (3)  identifies  issues 
that  libraries  should  consider  while  undertaking  network  statistics  and  measures  collection  and  reporting 
activities. 
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1.3.1  The  Network  Component  Perspective  as  a Framework  for  Developing  and 
Selecting  Network  Statistics  and  Measures 

The  Network  Component  Perspective  serves  as  a useful  mechanism  through  which  to  select  and 
measure  specific  aspects  of  a library’s  networked  services  and  resources  along  a number  of  network 
aspects. 

As  first  described  by  Bertot  and  McClure  (1996),  and  validated  throughout  other  research  studies,  the 
Network  Component  Perspective  provides  a framework  for  the  development  of  electronic  statistics  and 
performance  measures  (see  Figure  1.1).  This  perspective  suggests  that  there  are  a number  of  measurable 
components  to  electronic  networks: 

• Technical  infrastructure:  The  hardware,  software,  equipment,  communication  lines,  and  technical 
aspects  of  the  network  (e.g.,  workstations,  modems,  servers); 

• Information  content:  The  information  resources  available  on  the  network  (e.g.,  local  government 
information,  special  collections,  JSTOR,  ScienceDirect); 

• Information  services:  The  activities  in  which  users  can  engage  and  the  services  that  users  may 
use  to  complete  various  tasks  (e.g.,  online  reference  services,  usage  of  digital  information 
content); 

• Support:  The  assistance  and  support  services  provided  to  help  users  better  use  the  network  (e.g., 
training,  help  desk); 

• Management:  The  human  resources,  governance,  planning,  and  fiscal  aspects  of  the  network 
(e.g.,  network  staff,  advisory  boards,  budgeting). 

These  network  components  provide  a means  through  which  to  consider  the  type  of  statistics  and 
performance  measures  that  would  enable  research  libraries  to  describe  and  evaluate  their  networked 
services  and  resources. 

It  is  possible  to  look  at  the  information  content  aspect  of  a network  and  consider  network  statistics  and 
performance  measures  for  that  aspect  of  library  networked  services,  thus  enabling  one  to  map  the  network 
statistics  and  performance  measures  as  presented  in  Part  4 of  this  report. 

The  notations  in  parentheses  here  and  in  Figure  1.1  refer  to  the  statistics  and  measures  presented  in 
detail  in  Part  4 of  this  report. 

• Technical  Infrastructure 


(D3)  Cost  of  digital  collection  construction  and  management; 

• Information  Content 

(Dl)  Size  of  digital  library  collection; 

(PI)  Percentage  of  electronic  reference  of  total  reference; 
(P2)  Percentage  of  virtual  library  visits  of  all  library  visits; 
(P3)  Percentage  of  electronic  books  to  all; 

(Rl)  Number  of  electronic  full-text  journals; 

(R2)  Number  of  electronic  reference  sources; 

(R3)  Number  of  electronic  books; 
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(U5)  Number  of  virtual  visits  to  the  networked  library  resources; 

• Support 

(D3)  Cost  of  digital  collection  construction  and  management; 

• Management 

(Cl)  Cost  of  electronic  full-text  journals; 

(C2)  Cost  of  electronic  reference  sources; 

(C3)  Cost  of  electronic  books; 

(C4)  Library  expenditures  for  bibliographic  utilities,  networks,  and  consortia; 

(C5)  External  expenditures  for  bibliographic  utilities,  networks,  and  consortia; 

(D3)  Cost  of  digital  collection  construction  and  management. 

Using  this  approach,  libraries  can  gain  a sense  of  the  use  and  uses,  management,  and  reach  of  their 
networked  services  and  resources  in  specific  areas  or  across  a number  of  areas.  In  addition,  libraries  can 
look  in-depth  at  particular  aspects  of  their  networked  services  and  resources.  Moreover,  by  selecting 
statistics  and  performance  measures  relevant  and  of  interest  to  library  staff  and  managers  required  for 
reporting  purposes,  or  other  motivational  factors,  it  is  possible  for  libraries  to  develop  an  overall  sense  of 
their  networked  services  and  resources  along  a network  dimension.  The  Network  Component  Perspective 
also  serves  as  a framework  for  the  development  of  additional  network  statistics  and  measures  not  field- 
tested  during  this  project. 

Conceptualization  of  these  components  will  depend  on  the  particular  circumstances  of  the  individual 
library.  The  Network  Component  Model  (Bertot  and  McClure,  1998)  serves  as  a framework  for  such  a 
conceptualization  (see  Figure  1.2): 

• Extensiveness : How  much  service  the  library  provides  (e.g.,  number  of  users  accessing  a Web 
page  per  week,  number  of  database  sessions); 

• Efficiency : The  use  of  resources  in  providing  or  accessing  networked  information  services  (e.g., 
cost  per  session  in  providing  access  to  remote  users  of  an  online  database,  average  number  of 
times  users  are  unable  to  successfully  connect  to  the  library’s  servers); 

• Effectiveness : How  well  the  networked  information  service  meets  the  objectives  of  the  provider 
or  the  user  (e.g.,  success  rate  of  identifying  and  accessing  the  information  needed  by  the  user); 

• Service  quality : How  well  a service  or  activity  is  done  (e.g.,  percentage  of  transactions  in  which 
users  acquire  the  information  they  need); 

• Impact : How  a service  made  a difference  in  some  other  activity  or  situation  (e.g.,  the  degree  to 
which  network  users  enhanced  their  ability  to  gain  employment  or  pursue  business); 

• Usefulness : The  degree  to  which  the  services  are  useful  or  appropriate  for  individual  users  (e.g., 
percentage  of  services  of  interest  to  different  types  of  user  audiences); 

• Adoption : The  extent  to  which  institutions  or  users  integrate  and  adopt  electronic  networked 
resources  or  services  into  organizational  or  individual  activities  (e.g.,  answering  reference 
questions,  generating  inter-library  loan  requests,  use  of  digital  collections). 
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Figure  1.2  The  Network  Component  Model 
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It  may  not  be  necessary  to  develop  measures  that  address  every  evaluation  criteria  for  every  network 
component.  Instead,  the  matrix  serves  as  an  organizational  tool  for  evaluating  possible  measures  that  an 
individual  library  will  need  to  assess  its  outcomes  of  interest.  These  types  of  criteria  provide  an  important 
roadmap  for  thinking  about  the  type  of  data  element  and  statistics  that  would  be  necessary  to  produce  such 
measures,  as  well  as  providing  a quality  measurement  framework  for  library  networked  services  and 
resources. 

•v 

1.3.2  Using  the  Network  Statistics  ? 

The  recommended  network  statistics  and  performance  measures,  either  independently  or  in  some 
combination,  can  assist  research  libraries  in  describing  a number  of  aspects  of  their  networked  resources 
and  services.  This  section  provides  libraries  with  some  guidance  regarding  the  use  to  which  the  statistics 
and  measures  can  be  put. 

Although  the  statistics  and  measures  presented  in  this  manual  fall  under  a number  of  network 
components,  it  is  possible  to  categorize  broadly  the  statistics  and  measures  into  statistics  and  measures 
that  identify: 

1.  The  overall  size/volume  of  available  networked  resources; 

2.  The  extent  to  which  the  networked  resources  and  services  are  used  by  the  library’s  service 
population; 

3.  The  cost  to  the  library  of  providing  the  networked  resources  to  its  service  population. 

1.3. 2.1  Size  and  Volume  of  Networked  Resources 

As  Figure  1.3  indicates,  several  statistics  and  measures  allow  libraries  to  identify  the 
extensiveness  of  their  networked  resource  collections.  By  collecting  and  reporting  the  recommended 
statistics,  libraries  are  able  to  identify  their  total  number  of: 

• Journals  available  electronically  in  full-text  format  (e-joumals); 

• Reference  sources  available  electronically  to  staff  and  patrons; 

• Books  available  electronically  (e-books); 

• Items  digitized  from  the  library’s  own  collection  (e.g.,  documents  digitized  and  mounted  through 
a network  accessible  method). 
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Also,  by  combining  R 3 (Number  of  electronic  books)  with  a count  of  library  print  monographs, 
libraries  can  calculate  the  overall  percentage  of  books  available  in  electronic  format  to  the  populations  that 
they  serve  (P3). 

Figure  1.3  Network  Resources  Size  and  Volume  Statistics  and  Measures 

R1  Number  of  electronic  full-text  journals 
R2  Number  of  electronic  reference  sources 

R3  Number  of  electronic  books 

D 1 Size  of  library  digital  collection 

P3  Percentage  of  electronic  books  to  all  monographs 


1.3. 2.2  Extent  of  Use  of  Networked  Resources  and  Services 

A number  of  statistics  and  measures  allow  libraries  to  determine  the  overall  use  of  their  networked 
resources  and  services  (see  Figure  1.4).  In  particular,  statistics  enable  libraries  to  identify  the  overall: 

• Number  of  sessions  conducted  with  library  database  subscription  services,  with  a number  of  sub- 
categories if  desired  (e.g.,  location,  per  title,  etc.); 

• Number  of  queries  posed  by  users  of  the  database  subscription  services,  with  a number  of  sub- 
categories if  desired  (e.g.,  location,  per  title,  etc.); 

• Number  of  items  requested  (e.g.,  printed,  e-mailed,  saved,  or  otherwise  accessed)  by  users  of  the 
database  subscription  services,  with  a number  of  sub-categories  if  desired  (e.g.,  location,  per  title, 
etc.); 

• Number  of  visits  to  library-maintained  virtual  resources  (e.g.,  web  pages),  with  a number  of  sub- 
categories if  desired  (e.g.,  location  - remote  v.  in-library); 

• Number  of  accesses  and  queries  made  by  library  service  users  of  library-maintained  virtual 
resources,  with  a number  of  sub-categories  if  desired  (e.g.,  location  - remote  v.  in-library); 

• Number  of  electronic  reference  transactions  conducted. 

By  considering  the  U 1 (Number  of  electronic  reference  transactions)  and  U5  (Virtual  visits)  network 
statistics  and  traditional  statistics  of  reference  transactions  and  library  visitors,  libraries  can  gain  a sense  of 
the  ratio  of  electronic  reference  to  total  reference  transactions,  as  well  as  the  ratio  of  virtual  (remote) 
library  visits  to  physical  library  visits.  These  provide  important  trend  indicators  for  the  use  of  selected 
networked  library  resources  and  services. 

Figure  1.4  Extent  of  Use  of  Networked  Resources  and  Services  Statistics  and  Measures 

U 1 Number  of  electronic  reference  transactions 

U2  Number  of  logins  (sessions)  to  electronic  databases 

U3  Number  of  queries  (searches)  in  electronic  databases 

U4  Items  requested  in  electronic  databases 

U5  Virtual  visits 

D2  Use  of  library  digital  collection 

P 1 Percentage  of  electronic  reference  transactions  of  total  reference 

P2  Percentage  of  virtual  library  visits  of  all  library  visits 
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1.3.2.3  Library  Network  Resources  and  Services  Costs 

The  statistics  summarized  in  Figure  1.5  enable  research  libraries  to  identify  and  explore  a number  of 
costs  associated  with  developing,  acquiring,  and  managing  library  networked  resources  and  services.  In 
particular,  the  statistics  provide  libraries  with  cost  data  that: 

• Determine  the  cost  of  subscribing  to  and/or  purchasing  online  full-text  journals; 

• Determine  the  cost  of  subscribing  to  and/or  purchasing  online  reference  sources; 

• Determine  the  cost  of  subscribing  to  electronic  books  (e-books); 

o Identify  library  expenditures  for  online  material  provided  through  participation  in  local,  regional, 
and/or  national  consortia  not  included  in  database  subscription  services; 

• Identify  the  expenditures  of  non-library  entities  (e.g.,  state-wide  resources  provided  through 
consortia  or  governmental  agencies)  on  behalf  of  libraries  for  subscription  services; 

• Identify  the  costs  associated  with  library  digitization  activities,  including  equipment,  software, 
contracted  services,  personnel,  etc. 

As  a whole,  the  cost  statistics  provide  research  libraries  with  data  that  describe  some  of  the  direct  costs 
associated  with  providing  networked  resources  and  services  to  the  communities  they  serve.  : v 

Figure  1.  5 Library  Networked  Resources  and  Services  Cost  Statistics  and  Measures 

C 1 Cost  of  electronic  full-text  journals 
C2  Cost  of  electronic  reference  sources 

C3  Cost  of  electronic  books 

C4  Library  expenditures  for  bibliographic  utilities,  networks,  and  consortia 

C5  External  expenditures  for  bibliographic  utilities,  networks,  and  consortia 

D3  Cost  of  digital  collection  construction  and  management 


1.3.2.4  Combining  the  Recommended  Statistics  and  Performance  Measures 

While  it  is  possible  to  collect  and  use  the  statistics  independently,  it  is  in  the  combination  of  the 
statistics  that  libraries  can  derive  the  most  benefit  in  terms  of  understanding  the  breadth,  depth,  and  uses 
of  their  networked  resources  and  services.  As  an  example,  take  the  following  statistics: 

• R1  - Number  of  electronic  full-text  journals; 

• U2  - Number  of  logins  (sessions)  to  electronic  databases; 

• U3  - Number  of  queries  (searches)  to  electronic  databases; 

• U4  - Number  of  items  requested  in  electronic  databases; 

• Cl  - Cost  of  electronic  full-text  journals. 

By  collecting  all  five  (5)  of  the  statistics  and  viewing  the  results  in  combination,  libraries  can  begin  to 
develop  a comprehensive  view  of  their  online  subscription  services.  More  specifically,  libraries  would 
know: 


• How  many  full-text  journals  are  available  to  library  users; 

• How  many  times  users  access  the  online  journals; 
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• How  many  searches  users  conduct  while  using  the  online  journals; 

• How  many  articles  users  view  (broadly  defined  to  include  viewing,  printing,  saving,  or  other 
action)  from  the  journal;  and 

• What  it  costs  the  library  to  provide  users  with  access  to  full-text  journal  services. 

Together  these  statistics  serve  as  indicators  that  can  facilitate  such  library  resource  allocation  decisions  as: 

• Given  the  overall  use  of  online  full-text  journals,  does  the  library  need  to  augment  and/or  modify 
its  subscription  services? 

• Given  the  overall  cost  of  full-text  subscription  services,  does  the  aggregate  use  of  these  services 
justify  the  expenditure  when  compared  with  other  library  collection  and  resource  expenditures? 

• Are  there  discemable  usage  patterns  that  would  indicate  additional  resource  needs,  in  terms  of 
content  areas,  time  of  use,  frequency  of  use,  and  user  training  needs? 

• Does  the  library  hold  material  related  to  its  users’  needs  that  the  library  could  digitize,  organize, 
and  make  available  (for  example,  creating  finding  aids,  digitizing  unique  materials,  etc.)? 

Used  in  this  way,  these  statistics  provide  libraries  with  the  ability  to  review  online  full-text  subscription 
expenditures  given  the  overall  and  specific  uses  by  the  communities  served. 


A review  of  the  issues  and  perspectives  surrounding  statistics  and  performance  measures  point  to  a 
number  of  considerations  for  libraries: 

♦ Library  culture  of  assessment : In  addition  to  having  a systematic  approach  to  network  statistics 
and  performance  measure  activities,  libraries  need  to  adopt  an  overall  culture  of  assessment. 

Lakos  defines  a culture  of  assessment  as  (1999,  p.  5): 

The  attitudinal  or  institutional  changes  that  have  to  occur  in  order  for 
library  staff  to  be  able  to  work  in  an  environment  where  decisions  are 
based  on  facts,  research  and  analysis,  and  services  are  planned  and 
delivered  in  order  to  maximize  positive  outcomes  and  impacts  for  the 
library  clients. 

As  such,  libraries  need  to  focus  on  a systematic  approach  to  the  assessment  of  library  services, 
resources,  and  initiatives  in  order  to  better  understand  the  impact  of  those  services,  resources,  and 
initiatives  as  well  as  to  undertake  changes  and/or  modifications  to  best  meet  the  needs  of  library 
users.  While  several  participating  libraries  undertake  a number  of  assessment  activities,  they  are 
not,  in  general,  part  of  a systematic  evaluation  and  assessment  process  that  permeates  the  library. 
It  is  within  such  an  assessment  framework  that  network  statistics  and  performance  measurement 
activities  need  to  reside. 

• Library  data  collection,  analysis,  and  presentation  management  system:  The  field  test,  and 
various  research  activities  throughout  the  project,  demonstrated  that  libraries  overall  do  not 
engage  in  a systematic  and  focused  data  collection  system.  Few,  if  any,  have  staff  specifically 
responsible  for  library-wide  data  collection,  analysis,  or  reporting  and  presentation  efforts. 
Moreover,  library  staff  may  not  be  trained  appropriately  in  the  various  methodologies,  data 
analysis  techniques,  and  reporting  procedures  required  to  engage  in  statistical  and  performance 
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measurement  activities.  The  data  collection  situation  is  particularly  problematic  in  the  networked 
environment,  as  data  collection  efforts  in  this  area  require  additional  technical  and  research  skills. 

• Library  staff  development  and  training : Given  the  culture  of  assessment  issue,  combined  with  a 
general  lack  of  systematic  network  statistics  and  performance  measurement  activities  in 
participating  libraries,  it  is  clear  that  there  is  a need  for  staff  development  and  training  in  both 
assessment  and  network  statistics  activities.  This  training  should  incorporate  an  overview  of  the 
benefits  and  impacts  of  evaluation  activities;  the  value  of  evaluation  in  decision-making  and 
resource  allocation  processes;  network  statistics  and  performance  measure  definitions,  collection 
activities,  methodologies,  and  reporting  systems;  and  the  incorporation  of  findings  regarding 
network  statistics  and  measures  into  decision-making  and  resource  allocation  activities. 

• Network  planning  and  evaluation  activities  as  part  of  a larger  context:  The  perspective  presented 
in  Figure  1.1  of  this  section  demonstrates  clearly  that  research  library  network  activities  reside  in 
a larger  organizational  context.  Despite  varying  local  factors,  it  is  important  to  consider  the 
planning  and  evaluation  of  library  networked  resources  and  services  as  part  of  larger 
organizational  planning  and  evaluation  activities.  For  example,  the  decision  to  subscribe  to 
various  online  databases  needs  to  occur  in  the  larger  context  regarding  library  collection 
development  efforts. 

• Development  of  multi-agency  reporting  systems : It  is  clear  that  libraries  do  not  control  the  use 
data  for  all  networked  services  and  resources.  This  is  clearly  demonstrated  by  online  database 
vendor  statistics.  Libraries  that  receive  networked  services  and  resources  from  other  entities  such 
as  state  library  agencies  or  regional  consortia,  however,  also  need  to  work  with  those 
administrative  entities  for  usage  reports  as  well  to  get  a better  sense  of  the  overall  use  of,  for 
example,  database  services  from  a research  library  perspective.  In  such  cases,  usage  reports  will 
go  from  the  vendor  directly  to  the  subscribing  entity  - not  necessarily  all  the  participating 
members.  Thus,  there  is  a need  to  develop  a reporting  structure  that  goes  beyond  the  research 
library  in  such  cases.  It  is  also  necessary  to  construct  agreements  to  encourage  individual  library 
statistics  and  work  with  consortia  groups  to  generate  meaningful  reports  for  members,  etc. 

• Investment  and/or  modifications  in  infrastructure : Network  statistics  and  performance  measures 
are  dependent  on  the  information  technology  (IT)  architecture  of  a library,  consortia,  vendor,  or 
other  service  provider  from  which  the  library  derives  service.  If  a library  finds  certain  statistics  of 
interest  and/or  importance,  it  may  require  investment  in  an  IT  infrastructure  that  enables  the 
collection  of  such  statistics.  That  investment  may  require  IT  and  library  staff  training  in  order  to 
understand  the  configuration  as  well  as  the  statistics  enabled  through  such  a configuration. 

• Relationship  of  network  statistics  to  ARL  statistics  at  large : This  study  focused  on  network 
statistics  and  performance  measure  for  ARL  libraries.  It  did  not  consider  the  larger  ARL  statistics 
and  data  collection  context.  It  is  clear,  however,  that  there  is  a relationship  between  this  project 
and  the  ARL  statistics  program  in  general.  Indeed,  it  is  possible  to  consider  performance 
measures  that  combine  currently  collected  ARL  statistics  (e.g.,  volumes)  and  proposed  network 
statistics  (e.g.,  R3  - Number  of  electronic  books).  While  the  combining  of  traditional  and 
network  statistics  requires  additional  exploration,  ARL  and  ARL  members  will  need  to  consider 
how  the  proposed  network  statistics  fit  into  the  larger  ARL  statistics  program. 

• Relationship  of  network  statistics  to  other  ARL  research  initiatives  and  activities : Although  not 
explored  fully  by  the  study  team  to  date,  it  is  important  to  consider  the  relationship  that  the 
proposed  network  statistics  and  other  ARL  research  assessment  and  evaluation  initiatives  may 
hold.  Consider  the  network  statistics  and  the  LibQUAL+™  initiative.  It  is  conceivable  that  a 
library  could  engage  in  a combined  assessment  program  with  the  network  statistics,  which  serve 
to  describe  the  use  and  uses  of  particular  networked  services,  and  LibQUAL+™,  which  serves  to 
explore  user  satisfaction  with  those  services.  For  example,  libraries  could  collect  and  analyze  the 
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uses  of  their  online  database  services  (sessions,  searches,  items  requested).  Simultaneously, 
libraries  could  administer  a user-based  survey  to  users  of  the  database  services  and  explore  their 
satisfaction  with  those  services  in  general  (or  a particular  one).  Together,  these  assessment  efforts 
would  provide  libraries  with  detailed  use  and  satisfaction  data  regarding  a library’s  online 
database  services. 

• Relationship  of  network  statistics  to  other  data  collection  activities : With  the  number  of 

initiatives  in  the  library  network  statistics  area  identified  in  section  1 .4.1  of  this  report,  it  is  likely 
that  many  library  statistics  programs  and  data  collection  efforts  will  review  and  incorporate 
network  statistics  into  the  existing  data  collection  and  reporting  systems.  For  example,  the  State 
Library  Agency  survey — reported  and  conducted  by  the  National  Center  for  Education  Statistics 
(NCES)  — will  begin  reporting  online  database  usage  statistics  as  defined  by  Bertot,  McClure,  and 
Ryan  (2001)  with  their  2002  survey.  The  public  library  community,  again  through  annual  surveys 
conducted  in  cooperation  with  state  library  agencies,  the  U.S.  National  Commission  on  Libraries 
and  Information  Science,  and  NCES,  are  reviewing  what  network  statistics  to  collect.  More 
importantly,  through  NCES,  the  Integrated  Postsecondary  Education  Data  System  (IPEDS) 
collects  several  statistics  regarding  academic  library  holdings.  Under  consideration  now  are  what 
network  statistics  to  collect  through  that  survey.  In  an  environment  full  of  consortia  and  multi- 
type network  resource  and  service  sharing  — particularly  with  respect  to  online  database 
services — it  is  important  to  consider  what  network  statistics  are  common  across  libraries  in  order 
to  foster  comparability  and  reduce  data  collection  burdens  faced  by  reporting  institutions. 

These  issues  point  to  the  need  for  additional  research  and  testing  of  a measurement  framework  for 
network  resources  and  services,  as  well  as  the  relationship  of  the  network  statistics  to  other  research 
initiatives.  The  perspective  presented  in  Figure  1.1  serves  as  a beginning  point  for  such  a framework. 

The  issues  raised  also  indicate  a need  for  libraries  to  enhance  technical,  evaluation,  planning,  and 
methodological  skills  among  librarians  to  better  understand  evaluation  activities  in  general  and  the 
networked  environment  in  particular. 
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1.4  Outcomes 

It  is  not  enough  simply  to  develop  measures  and  collect  statistics  related  to  library  networked 
resources  and  services.  Indeed,  as  state  legislatures  increasingly  tie  budgets  to  performance  and  regional 
accreditation  boards  begin  emphasizing  the  need  to  articulate  outcomes,  it  is  important  for  research 
libraries  to  decide  what  their  outcomes  should  be  and  to  determine  how  to  connect  measures  and  statistics 
to  these  outcomes.  An  important  component  of  this  project  involves  developing  a graphical  model  that 
ultimately  links  proposed  network  statistics  and  measures  to:  (1)  educational,  research,  and  service 
outcomes  in  higher  education  institutions;  and  (2)  educational,  research,  and  service  outcomes  in  higher 
education  libraries.  For  “outcome”  we  have  used  the  following  as  a working  definition: 

a clearly  identified  result  or  end  product  that  occurs  as  a consequence  of 
individual  or  combined  activities  from  units  at  the  institution.  These 
outcomes  are  a preferred  or  desired  state  and  clarify  specific  expectations 
of  what  should  be  products  from  the  institution.  An  institutional  outcome 
can  be  defined  and  measured  in  such  a way  that  evidence  is  available  to 
determine  the  amount  or  degree  to  which  the  outcome  does,  in  fact, 
occur. 

As  part  of  the  overall  project,  the  study  team  has  undertaken  a number  of  outcomes-related  activities  to 
date. 


1.4. 1 Overview  of  Outcomes  focus 
1.4. 1.1  Review  of  Outcomes-Related  Project  Team  Efforts  to  Date 

During  Phase  II  of  the  E-Metrics  project  (November  2000  - June  2001),  the  study  team  was  engaged 
in  an  extensive,  ongoing  effort  to  develop  frameworks  for  understanding  and  graphical  models  for 
depicting  measurable  library  inputs  and  outputs  in  the  context  of  indicators  of  institutional  outcomes. 

This  process  began  with  a review  of  literature  concerning  library  and  institutional  assessment  to  better 
understand  what  work  and  thinking  had  been  done,  with  an  eye  toward  finding  documented  linkages 
between  library  outputs  and  institutional  outcomes.  We  found  that,  while  the  problem  was  clearly  defined 
and  its  significance  well  appreciated  and  often  noted  in  the  literature,  there  has  been  little  work  toward 
identifying  linkages  and  developing  models  that  ARL  member  libraries  could  use  to  determine  how  to 
best  measure  their  impact  on  the  outcomes  of  the  universities  they  support. 

Our  earlier  Discussion  Paper:  Towards  a Framework  of  Library  and  Institutional  Outcomes , dated 
and  distributed  April  4,  2001  [available  at 

http://www.ii. fsu.edu/Proiects/ARL/Docs/ARL.Outcomes.Disc.Paper.CNI.April4C. 2001. dock  includes  a 
literature  review  and  highlights  some  of  the  key  work  identified  in  the  area  of  outcomes  assessment. 

From  the  insights  gained,  in  addition  to  prior  work  in  this  area  by  the  team,  we  conducted  selective 
structured  interviews  to  begin  the  process  of: 

1 . identifying  key  work  and  actors  in  the  field; 

2.  grounding  our  framework  development  in  (a)  the  activities  already  underway,  and  (b)  the 
concerns  of  representative  participants. 
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In  addition,  the  study  team  has  been  and  is  engaging  in  a multi-method  approach  that  creates  many 
access  points  to  the  complex  issues  at  hand,  including: 

• Content  analysis  of  ARL  member  institution  strategic  planning  documents; 

• A survey  of  ARL  deans  and  directors; 

• Discussion  forums; 

• Site  visits; 

• Policy  analyses; 

• Case  studies. 

Ongoing  content  analysis  of  surveys,  interviews,  and  strategic  planning  documents  provided  to  us  by 
survey  respondents  and  others  has  begun  to  reveal  commonalities  in  the  institutional  goals  of  ARL 
members  and  the  manner  in  which  those  goals  are  devised  and  articulated.  Synthesis  of  these  documents 
has  informed  the  refinement  of  our  model  by  suggesting  a set  of  key  processes  that  ARL  deans  and 
directors  can  employ  to  ensure  their  libraries  are  having  the  desired  outcomes  and  that  they  are  able  to 
measure  them  successfully.  The  study  team  has  conducted  site  visits  that  furthered  our  understanding  of 
the  cultural  context  in  which  outcomes  assessment  must  take  place,  and  provided  an  opportunity  for 
model  testing  and  refinement. 

Upon  completion,  we  hope  to  have  a framework  and  model  that  are  valid  for  the  ARL  community  and 
that  can  be  built  upon  in  future  research.  A proposal  for  external  funding  of  such  research  will  be  written 
by  the  study  team  so  that  the  contribution  the  library  makes  to  the  institution  it  serves  can  be  better 
measured,  articulated,  and  enhanced. 

1.4.1. 2 Development  of  an  Outcomes  Model 

Lindauer’s  (1998)  “Defining  and  Measuring  the  Library’s  Impact  on  Campuswide  Outcomes,” 
particularly  informed  the  graphical  model  presented  in  this  project’s  Discussion  Paper.  In  this  article, 
Lindauer  describes  five  assessment  domains  for  libraries  and  connects  goals  of  the  university  with 
activities  and  measures  within  the  library.  Our  graphical  model  began  to  depict  the  process  by  which  an 
academic  research  library  helps  meet  the  goals  of  particular  departments  and  functional  units  within  a 
university,  which  in  turn  contribute  to  institutional  goals,  while  acknowledging  that  libraries  may  also 
contribute  more  directly. 

Our  research  efforts — particularly  site  visits,  structured  interviews,  and  strategic  planning  document 
content  analysis — have  demonstrated  the  need  for  libraries  to  approach  measuring  outcomes  differently 
than  measuring  inputs  and  outputs.  The  current  graphical  model  (Figure  1 .6)  shows  the  manner  in  which 
library  deans  and  directors  must  be  cognizant  of  the  objectives  and  goals  of  the  university,  other  academic 
and  functional  units,  and  the  unique  and  changing  legislative,  accreditation,  and  technological 
environments  in  which  universities  operate. 

While  the  graphical  model  does  not  show  how  particular  services  and  resources  contribute  to 
outcomes,  it  does  describe  how  a library  dean  or  director  can  begin  thinking  about  targeting  library 
services  toward  the  outcomes  of  interest  to  the  university,  and  measure  their  contributions  and  impacts 
more  effectively.  Linking  measures  to  outcomes  cannot  be  done  without  first  considering  what  outcomes 
are  desired  and  what  effects  achieving  those  outcomes  could  have. 
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Figure  1.6  Outcomes  Model 
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1.4.2  Relationship  between  Outcomes  and  Measurements 
1.4.2.1  A Change  in  Perspective 

University  research  libraries  are  established  to  support  the  broad  research,  education,  and  service 
goals  that  are  fundamental  to  the  mission  of  the  institutions  they  serve.  Beyond  helping  to  fulfill  the 
university  mission,  a research  library  must  be  able  to  help  the  larger  institution  reach  its  more  concrete  but 
shifting  goals.  These  goals  may  be  articulated  in  strategic  planning  documents,  in  conversations  with  the 
provost  or  academic  deans,  in  fulfilling  regional  accreditation  standards,  or  in  state  legislation. 

It  is  important  to  be  highly  aware  of  these  goals  and  to  be  able  to  target  library  resources,  services, 
and  programs  to  help  meet  institutional  goals.  Doing  so  is  critical  to  the  university  research  library  being 
seen  as  a vital,  contributing  part  of  the  university.  To  address  these  goals  and  measure  them  effectively,  it 
is  important  to  ask  three  key  questions: 

1 . What  is  the  desired  state  of  the  university? 

2.  How  can  the  library  help  the  university  achieve  this  state? 

3.  How  will  the  library  know  when  it  has  been  successful  in  helping  the  library  achieve  this  state? 

The  first  question  helps  the  library  better  understand  its  operating  environment.  Knowing  what  the 
university  wants  to  be  now  and  in  the  future  helps  library  administrators  understand  what  customers  need. 
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The  second  question  helps  in  making  decisions  about  which  of  the  many  goals  of  the  university  are 
helpful  for  the  library  to  focus  on.  Also,  answering  this  question  helps  the  library  make  decisions  about 
what  action  it  must  take  to  contribute  to  the  fulfillment  of  those  goals.  Existing  services  and  programs 
may  address  them,  programs  may  have  to  be  fine-tuned  and  resources  upgraded,  or  new  programs  and 
resources  may  need  to  be  developed  and  acquired. 

The  third  question  helps  the  library  craft  the  measures  that  will  provide  indications  of  success.  This  is 
extremely  important  and  must  be  thought  about  carefully  because  direct  outcomes  measurement  is  often 
difficult  or  impossible.  It  may  be  necessary  to  develop  several  measures  that  work  together  to  indicate  an 
outcome  state  or  to  use  surrogate  measures  such  as  perception  surveys.  One  framework  for  developing 
such  measures  is  described  below. 

The  order  in  which  these  questions  are  addressed  is  also  vital.  Before  measurements  can  be  derived, 
library  administrators  must  know  what  they  want  to  measure  and  how  the  library’s  impact  is  likely  to 
occur.  Without  this  perspective  and  focus,  outcomes  measurement  cannot  be  targeted  and  loses  meaning. 
In  other  words,  it  is  very  difficult  to  gain  useful  insights  about  outcomes  when  measures  are  not  designed 
with  outcomes  explicitly  in  mind. 

ARL  libraries  may  currently  collect  measures  that  can  provide  some  indication  of  success  of  a 
particular  program  or  service  provided  to  customers,  such  as  user  satisfaction  surveys;  however,  it  is 
important  to  think  broadly — with  the  desired  state  in  mind — and  not  simply  use  the  measures  on  hand 
because  they  are  easy  to  collect  or  because  a lot  of  time  and  effort  has  been  devoted  to  collecting  them. 

In  most  cases,  a single  measure  on  its  own  is  not  enough  to  indicate  whether  a research  library  is 
successful  in  a given  area.  To  accurately  assess  the  success  or  quality  of  an  academic  library, 
measurement  should  be  implemented  at  three  key  levels: 

• Outcome  Level; 

• Use/Capacity  Level  (Output); 

• Resource  level  (Input). 


Figure  1.7  Using  Measures  to  Answer  Questions  at  Different  Levels 


Outcome  Level 

Use/Capacity  Level 
(Output  Measures) 

Resource  Level 
(Input  Measures) 

What  are  the  results  of  a 
program  or  process? 

How  much  is  a service,  resource, 
or  program  being  used? 

What  do  we  need  to  ensure  success? 

How  successful  or  effective  is 
the  library? 

Who  is  using  a service,  resource, 
or  program? 

What  funding  level  is  appropriate  or 
necessary  for  a particular  program? 

How  effective  do  customers 
perceive  your  programs  to  be? 

Why  are  people  using  a 
particular  program? 

Do  we  need  more  of  a particular 
resource  in  order  to  have  a more 
effective  program? 

What  beneficial  effects  are  you 
having  on  your  customers? 

How  could  a program  be 
changed  to  better  suit  the  needs 
of  your  customers? 

The  following  example  illustrates  this  approach: 

Imagine  that  information  literacy  is  a theme  for  your  university.  How 
can  you  have  a meaningful  impact  in  this  area?  One  way  might  be  an 
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effective  bibliographic  instruction  program  that  reaches  many  students. 

Ask  the  question,  “What  would  constitute  success  for  our  BI  program?” 

Once  you  have  articulated  the  answer,  you  can  begin  to  develop  measures 
for  the  effectiveness  of  your  BI  program  (i.e.,  professor  perception  of 
student  performance  before  and  after  BI  program,  etc.) 

It  is  also  important  to  know  how  many  students  can  be  reached  with  the  program,  so  capacity  and  use 
measures  are  needed  as  well.  Because  the  program  needs  staff  and  materials  created,  it  is  important  to 
have  resource  measures  that  indicate  what  goes  into  the  program  and  whether  more  or  different  resources 
make  a difference  in  the  effectiveness  or  success  of  your  program. 

An  example  in  the  networked  context  is  the  following: 

Imagine  that  your  university  wants  to  attract  a “world  class  faculty.” 

There  are  many  ways  that  the  library  can  have  an  impact  in  this  area. 

Capitalizing  on  resources  unique  to  your  university  is  one  way  to  “brand” 
your  library  (and  in  turn  university)  as  having  expertise  in  a particular 
field  (or  many  fields).  By  digitizing  collections  and  making  them  widely 
available,  the  library  can  attract  scholars  all  over  the  world  to  these 
materials.  As  a result,  scholars  will  begin  to  associate  these  unique 
resources  with  your  university  and  may  even  be  attracted  to  your  faculty. 

There  are  many  ways  to  measure  whether  or  to  what  extent  there  is  a 
linkage.  For  instance,  you  could  survey  new  faculty  members  and  ask  if 
they  used  your  library’s  materials  prior  to  coming  on  board,  which 
materials  they  used  and  for  what,  and  whether  this  influenced  their 
decision  to  join  the  faculty. 

In  this  example,  it  is  important  to  know  who  is  accessing  the  collection,  and  which  parts  of  the 
collection  they  are  using  most  often.  This  can  indicate  which  parts  the  digital  collection  users  are  most 
interested  in  and  what  collections  may  need  to  be  more  fully  developed.  Additionally,  it  would  be  helpful 
to  know  what  resources  are  devoted  to  creating  this  digital  collection,  and  if  increasing,  changing,  or 
upgrading  resources  has  an  impact  on  the  desired  outcome. 

Following  this  approach,  however,  may  lead  to  the  formulation  of  a wide  range  of  performance 
measures  and  statistics.  Selection  of  the  precise  measures  needed  to  evaluate  an  electronic  resource  or 
service  can  be  especially  difficult,  even  for  libraries  that  have  undertaken  processes  similar  to  those 
described  above.  Therefore,  it  is  important  to  have  a framework  to  assist  in  choosing  measures  to  gain 
insights  into  the  use  and  uses,  management,  and  reach  of  networked  services  and  resources  in  specific 
areas  or  across  a number  of  areas. 

1.4.2.2  Importance  of  Developing  Outcomes  Processes 

As  discussed  in  the  preceding  section,  ARL  libraries  need  to  develop  a process  to  identify  and 
operationalize  library  outcomes  that  contribute  to  institutional  outcomes.  The  library  must  play  a major 
role  in  informing  the  university  of  valued  institutional  outcomes  to  which  the  library  contributes.  Setting 
up  such  a process  is  an  important  method  for  informing  key  stakeholders  in  the  university  of  both  the 
library’s  role  in  institutional  outcomes  and  insuring  that  the  institutional  outcomes  to  which  the  library  has 
(or  may  have)  links  are  in  fact  appropriate. 
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Because  each  university  has  different  processes  for  information  sharing,  decision-making,  and 
mission  fulfillment,  it  is  important  that  each  university  library  identify,  understand,  and  master  the 
established  local  process.  It  is  a given  that  libraries  must  work  within  their  particular  organizational 
framework.  Therefore,  to  maximize  contributions  to  university  outcomes,  the  library  must  orient  itself 
within  and  operate  with  the  fullest  advantages  of  that  local  framework. 

An  important  factor  that  contributes  to  an  effective  understanding  of  the  local  situation  is  sensitivity  to 
the  differing  points  of  view  of  various  stakeholder  groups.  What  might  constitute  institutional  outcomes, 
or  appropriate  institutional  outcomes,  will  vary  from  one  group  to  another  (deans,  faculty,  trustees, 
students,  etc.).  It  may  be  that  the  outcomes  from  the  library  (as  agreed-upon  by  library  staff  and 
administration)  may  or  may  not  be  those  seen  by  university  administration  as  important  or  appropriate. 
Developing  a process  to  address  these  potentially  conflicting  stakeholder  concerns  at  the  local  level  is 
particularly  vital. 


1.4.3. 1 Why  measure? 

In  general,  statistics  and  performance  measures  can  inform  decision-makers  at  many  levels  in  the 
research  library  organization.  At  one  end  of  the  spectrum,  they  can  support  the  wide  array  of  internal 
operations;  at  the  other  end  they  can  provide  insights  on  the  broader  role  played  by  the  library  in  its 
institutional  and  community  setting.  More  specifically,  they  give  administrators  the  tools  to  know  how 
effectively  they  provide  resources  and  services  to  customers,  how  their  customers  perceive  their 
effectiveness,  and  how  well  they  are  doing  as  compared  to  peer  institutions. 

1.4.3. 2 Why  measure  Networked  Services? 

Although  libraries  have  changed  significantly  in  response  to  electronic  and  networked  services,  the 
ARL  membership  criteria  index  does  not  currently  contain  measures  that  pertain  to  all  of  these  services 
and  document  the  substantial  amount  of  time  and  money  involved.  It  does,  however,  include  measures 
that  address  electronic  resources  and  services  as  part  of  the  larger  whole.  Specifically,  serials  include 
electronic  journals  and  total  expenditures  includes  costs  for  electronic  services.  “Striking  the  right 
balance  between  measuring  the  continuing  and  the  emerging  realities  of  the  modem  research  library  is  at 
the  cornerstone  of  the  ARL  Statistics  and  Measurement  Program  operations.  Research  libraries’ 
traditional  realities  drive  the  ARL  measures  of  printed  collections,  budgets,  and  staffing.  The  emerging 
realities  drive  ARL’s  agenda  to  seek  credible  indicators  of  steady  growth  and  high  demand  for  the 
complex  mix  of  new  services,  consortial  arrangements,  electronic  information,  the  influence  of  the 
Internet,  and  the  ways  in  which  students  and  faculty  interact  with  each  other  and  these  newer  channels  of 
information”  (Kyrillidou  & Crowe,  1998). 

1.4.3.3  E-Metrics 

Overall,  the  network  statistics  and  measures  described  in  the  following  manual  were  developed, 
refined,  and  tested  to  help  describe  what  is  happening  internally  in  research  libraries  and  do  not  measure 
outcomes  directly.  They  may,  however,  serve  as  indicators  of  outcomes  or  point  to  areas  where  outcomes 
measures  could  be  focused.  Each  of  the  five  types  of  statistics/measures  can  be  helpful  in  a variety  of 
ways. 
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1.4.3. 3.1  Resource  Measures 

The  current  ARL  membership  criteria  index  does  not  include  separate  measures  for  electronic  and 
networked  monographs,  serials,  and  bibliographic  utilities,  but  does  include  some  of  them  with  their 
traditional  analogs.  However,  to  be  able  to  assess  the  relative  cost  and  benefits  of  materials  in  different 
formats,  as  well  as  for  the  reasons  pointed  out  in  Section  1. 3.4.2,  decoupling  the  measures  for  electronic 
and  traditional  materials  may  be  necessary. 

Although  the  same  can  be  said  of  much  of  what  is  on  the  Internet,  electronic  and  networked  resources 
provide  scholarly  and  other  materials  that  most  users  of  the  Internet  cannot  get  on  their  own  and  thus 
contribute  to  the  value  of  the  research  library.  The  number  can  also  be  benchmarked  against  other 
institutions  and  used  to  show  where  academic  research  libraries  are  in  terms  of  providing  the  most 
anytime-anywhere  access  to  scholarly  materials. 

1.4.3. 3. 2 Use  Measures 

High  use  of  a library  resource  or  service  implies  a collection  development  program  that  is  working  to 
create  access  to  the  resources  customers  need.  Use  measurements  can  also  identify  resources  and  services 
that  are  seen  as  particularly  valuable  in  the  education  and  research  enterprise  and  should  be  expanded,  or 
perhaps  resources  and  services  that  should  be  discontinued  due  to  lack  of  use  and  interest.  Whether 
provided  by  vendors  or  collected  internally,  usage  statistics  can  help  a library  administrator  make 
decisions  and  plan  for  the  future.  Looking  at  the  reported  data  can  also  provide  other  information  as  well, 
such  as  where  and  when  people  use  materials,  thus  informing  the  library  if  it  is  truly  providing  anytime- 
anywhere  access.  Providing  this  access  is  expensive  and  can  drain  funds  away  from  traditional  access. 
The  E-Metrics  use  measures  put  this  in  the  perspective  of  the  changing  research  library. 

1.4.3.3.3  Cost  Measures 

These  measures  were  developed  for  ARL  to  understand  what  libraries  were  spending  on  electronic 
and  networked  resources  because  “expenditures  for  electronic  resources  confirmed  that  existing  data 
[was]  neither  comprehensive  nor  comparable  from  one  library  to  another”  (Blixrud  & Jewell,  1998). 

These  measures  can  help  ARL  libraries  answer  questions  posed  to  them  by  university  administration  as 
they  increasingly  need  to  justify  their  growing  budgetary  needs  due  to  the  great  expense  of  electronic  and 
networked  services. 

1.4.3. 3. 4 Digital  Collection  Measures 

While  comprised  of  resource  and  user  measures,  the  digital  collection  measures  attempt  to  describe 
where  libraries  are  in  creating  and  making  available  local,  possibly  unique  content  that  may  not  have  been 
previously  network  accessible.  These  collections  are  important — and  in  some  cases  vital — for  users,  and 
help  to  brand  the  university  and  its  library  when  the  higher  educational  and  scholarly  community  uses 
them  and  notes  their  origin.  This  can  attract  students  and  faculty  to  your  university  as  well  as  adding  to 
the  world’s  networked  knowledge.  As  more  libraries  do  this,  more  users  will  be  able  to  get  specific, 
unique  resources  anytime  and  from  anywhere.  Collecting  these  measures  and  storing  them  centrally 
provides  an  opportunity  for  benchmarking  and  may  encourage  libraries  to  devote  more  time  and  resources 
to  this  worthwhile  endeavor. 
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1.4.3. 3. 5 Performance  Measures 

Performance  measures  are  useful  to  libraries  in  decision-making  pertaining  to  resource  allocation, 
service  offerings,  collection  development,  and  budgeting.  The  E-Metrics  performance  measures  are 
intended  to  help  libraries  know  and  describe  where  they  are  with  providing  electronic  networked  services 
and  resources,  as  compared  with  traditional  services  and  resources  in  order  to  “describe  the 
transformations  underway”  (Kyrillidou  & Crowe,  1998). 

There  is  also  a need  to  know  about  the  trends  across  institutions.  “The  challenge,  then,  is  not 
describing  any  single  change,  but  rather  to  develop  quantifiable  trend  analysis  in  multiple  institutions  that 
can  be  executed  from  year  to  year  in  a consistent  way”  (Kyrillidou  & Crowe,  1998). 

1.4.4  Ongoing  Efforts 

As  part  of  the  ongoing  focus  on  outcomes,  the  study  team  is  developing  a revised  version  of  the  April 
2001  Discussion  Paper , scheduled  to  be  completed  in  fall  2001.  Included  in  this  revision  will  be  an 
analysis  of  the  study  team’s  survey  of  ARL  deans  and  directors  related  to  institutional  outcomes.  Other 
aspects  of  this  ongoing  effort  follow. 

1.4.4. 1 The  Role  of  Regional  Accreditation  Standards 

The  study  team  engaged  the  services  of  Bonnie  Gratch-Lindauer  to  follow  up  on  some  of  her  previous 
work  (Lindauer,  1998),  and  conduct  a review  of  relevant  accreditation  standards  as  they  relate  to 
outcomes  and  networked  services.  Her  summary  observations  from  that  review  are  as  follows; 

1.  The  majority  of  these  outcomes  and  outcomes-related  statements  that  refer  to  libraries  and 
information  resources  are  located  in  sections  of  the  standards  that  deal  with  the  education  program 
and  institutional  effectiveness. 

2.  The  use  of  library  and  information  resources  is  connected  to  student  learning  outcomes  in  four  of 
the  documents,  and  evidence,  such  as  inclusion  in  course  syllabi  and  integration  of  library  use  into 
the  undergraduate  curriculum,  are  offered  as  measurable  indicators  for  assessment  purposes  in 
two  of  the  documents. 

3.  The  university  library’s  role  in  helping  students  develop  information  literacy  skills  is  an  important 
student  learning  outcome  referenced  in  four  of  the  documents  and  in  the  “Best  Practices  for 
Electronically  Offered  Degree  and  Certificate  Programs,”  endorsed  by  the  accrediting 
commissions. 

4.  Assessing  student  needs,  perceptions  and  levels  of  satisfaction  with  educational  support  services 
(i.e.,  library  and  information  services)  and  demonstrating  that  the  findings  from  these  user  studies 
are  used  for  program  improvement  are  fundamental  expectations  of  all  the  regional  accrediting 
commissions. 

5.  Appraisal  of  annual  institutional  goals  and  progress  in  their  accomplishment  is  suggested  as  a type 
of  evidence  contributing  to  institutional  outcomes,  or  in  some  of  the  documents  the  phrase  used  is 
“institutional  effectiveness.” 

6.  All  of  the  standards  describe  the  need  for  institutions  to  have  an  assessment  or  evaluation  plan  and 
to  document  that  the  findings  are  utilized  for  program  improvement.  Some  of  the  documents 
clarify  this  requirement  to  mean  that  each  program  or  unit  should  have  an  assessment  plan. 
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7.  Several  of  the  documents  refer  to  the  campus  climate  or  the  institutional  environment  that 

supports  teaching  and  learning.  Three  specifically  connect  library  and  information  resources  and 
services  to  the  quality  of  the  learning  environment.  The  implication  is  that  university  libraries 
should  clearly  describe  what  resources  and  services  they  provide  that  directly  support  the  learning 
environment,  how  these  are  used  and  with  what  effects  on  students  and  faculty. 

Her  findings  will  be  further  incorporated  into  the  next  version  of  our  Discussion  Paper  to  help  illustrate: 
(1)  the  need  for  research  university  libraries  to  demonstrate  the  outcomes  of  electronic  and  networked 
services;  (2)  the  need  for  such  libraries  to  demonstrate  any  outcomes  apart  from  electronic  and  networked 
services;  and  (3)  the  need  for  research  universities  to  show  the  connection  between  the  use  of  electronic 
and  networked  services  and  the  fulfillment  of  their  missions 

1.4.4. 2 Other  Outcomes-Related  Research 

The  study  team  has  also  been  reviewing  the  work  done  by  Ken  Smith  and  Doug  Jones,  who  are 
studying  education  and  research  outcomes  for  university  libraries,  respectively 
(http://www.arl.org/stats/newmeas/outcomes/heo.html).  It  is  hoped  that  their  work  will  help  us  better 
understand  the  role  electronic  and  networked  services  play  in  these  areas.  In  addition,  the  work  being 
done  to  study  user  perceptions  of  quality  in  the  LibQUAL+™  program  has  been  of  interest  and  is  being 
explored  for  possible  connections  (http://www.arl.org/libqual/). 

The  study  team  is  also  investigating  the  work  done  outside  of  ARL’s  New  Measures  Initiative.  One 
interesting  study  of  the  library’s  impact  on  sponsored  research  funding  conducted  by  Brinley  Franklin, 
Director  of  Library  Services,  University  of  Connecticut  Libraries,  and  Consulting  Associate,  KPMG 
Consulting,  LLC,  found  that  “electronic  services  use  supporting  sponsored  research  generally  mirrored  the 
same  level  of  support  exhibited  by  the  general  use  of  library  materials  and  services  at  almost  all  types  of 
libraries”.  Franklin  also  found  “a  high  correlation  between  total  research  and  development  funding  at  an 
educational  institution  and  total  library  expenditures  at  research  universities”.  This  work  suggests  that 
electronic  services  use  be  quantified  to  reflect  the  degree  to  which  a library’s  investment  in  electronic 
services  supports  specific  institutional  outcomes. 

It  is  hoped  that  the  work  described  above  will  help  us  better  understand  the  role  library  networked 
services  have  in  supporting  research,  education,  and  service  outcomes  at  a variety  of  ARL  member 
institutions. 


1.4.4.3  Proposal  for  Future  REsearch 

The  study  team  has  begun  thinking  about  how  best  to  move  forward  beyond  the  scope  of  this  project 
and  plans  to  release  a proposal  for  such  work  in  December  2001.  In  the  past,  it  has  been  said  that 
understanding  outcomes  will  require  much  research  and  indeed  this  is  consistent  with  the  study  team’s 
findings.  “In  the  long  run,  if  higher  education  wants  to  measure  library  ‘impact’  we  will  need  to  initiate 
longitudinal  studies,  for  example,  by  questioning  and  tracking  individuals  from  grades  K-12  through  their 
undergraduate/graduate  study  and  as  alumni  in  order  to  assess  how  their  lives  have  been  affected  by  their 
library  experience”  (Kyrillidou  & Crowe,  1998) 

1.5  Related  Efforts 

This  section  reviews  briefly  additional  research  initiatives  related  to  network  statistics  and 
performance  measurement  as  well  as  outcomes-based  research  in  the  higher  education  environment.  In 
particular,  the  section  identifies  and  describes  selected  initiatives  that  explore  the  development  and 
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implementation  of  network  statistics  and  performance  measures  across  library  types  as  well  as  initiatives 
that  attempt  to  link  the  research  library  to  institutional  goal  and  objective  attainment. 


Interest  in  the  development  and  implementation  of  network  statistics  and  performance  measures  has 
risen  steadily  over  the  last  three  years  in  a number  of  library  settings.  It  is  not  surprising,  therefore,  that  a 
number  of  library-based  organizations,  researchers,  publishers,  and  database  aggregators  pursue 
increasingly  the  ability  to  measure  network  activities  in  libraries.  Below  is  a selective  list  of  major 
initiatives,  other  than  the  ARL  project  on  which  this  document  reports,  active  in  the  area  of  library 
network  statistics  and  performance  measure  development: 

• Institute  of  Museum  and  Library  Services , National  Commission  on  Libraries  and  Information 
Science,  National  Information  Standards  Organization : Along  with  some  of  the  project  study 
team  members,  these  organizations  are  working  together  on  a number  of  initiatives  that  include 
the  development  and  adoption  of  network  statistics  for  U.S.  public  libraries,  the  development  and 
implementation  of  a national  data  collection  model  for  U.S.  public  library  network  statistics  (see 
http://www.ii.fsu.edu  for  additional  information),  the  adoption  of  standard  terminology, 
definitions,  and  reporting  of  database  vendor  statistics  (see  http://www.nclis.gov  for  additional 
information),  the  revision  of  U.S.  library  statistics  standards  (NISO  Z39.7,  see 
http://www.niso.org  for  additional  information)  to  include  network  statistics  and  performance 
measures,  and  the  adoption  of  international  standards  for  both  library  statistics  in  general  and 
network  statistics  in  particular  (see  below  for  a description  of  the  International  Standards 
Organization  initiative). 

• International  Coalition  of  Library  Consortia  (ICOLCf.  ICOLC  is  an  international  coalition  of 
predominantly  academic  libraries  (some  of  which  are  sponsors  of  the  ARL  E-Metrics  project) 
interested  in  pursuing  standard  network  statistics  and  reporting  systems  regarding  database  vendor 
data.  ICOLC  first  published  its  proposed  standards  and  definitions  in  November  1998  and  is 
currently  considering  revisions  to  those  standards.  Additional  information  on  the  ICOLC 
initiative  is  available  at  http://www.librarv.yale.edu/consortia/webstats.html. 

• International  Standards  Organization  (ISO)\  Through  the  ISO  Technical  Committee  46 
(Information  and  Documentation),  subcommittee  08  (Statistics  and  Performance  Evaluation) 
members  of  ISO  have  been  revising  both  general  library  statistic  standards  and  incorporating 
network  statistics  and  performance  measures  into  the  statistical  data  collection  efforts  of 
participating  libraries  (multi-type).  As  of  July  2001,  the  U.S.,  through  NISO,  rejoined  the  ISO 
effort  after  a one-year  absence.  A member  of  the  study  team  (Bertot)  serves  as  one  of  the  U.S. 
delegates,  as  well  as  an  ARL  library  director  (Heath).  Recent  balloting  efforts  resulted  in  the 
passage  of  the  proposed  ISO  library  statistics  (document  ISO/DIS  2789)  although  a number  of 
voting  members  provided  substantial  comments  on  the  statistics.  A meeting  will  occur  in  late 
August  2001  to  begin  resolving  the  issues  raised  by  members  through  their  comments.  Additional 
information  on  this  and  other  ISO  efforts  are  available  at 
http://www.iso.ch/iso/en/ISOOnline.frontpage. 

• European  Community-sponsored  Equinox  project:  The  Equinox  project  focused  on  developing 
library  performance  and  quality  measures.  In  particular,  the  project  aimed  to  further  develop 
existing  international  agreement  on  performance  measures  for  libraries  for  the  electronic  library 
environment  as  well  as  develop  and  test  an  integrated  quality  management  and  performance 
measurement  tool  for  library  managers.  The  project  identified  a number  of  performance  measures 
that,  in  some  cases,  have  been  integrated  into  the  ISO  library  statistics  initiative.  Additional 
information  on  the  Equinox  project  is  available  at  http://equinox.dcu.ie/. 
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• LibEcon  project : A European  initiative,  LibEcon  focuses  on  the  collection  of  economic  and  other 
library-related  data  from  predominantly  European  libraries.  For  its  Millennium  Study,  the  survey 
incorporated  selected  network  statistics  developed  by  the  IMLS  study  as  well  as  ISO  activities. 
The  US  is  reporting  for  the  first  time  in  2001.  NCLIS  is  coordinating  the  data  submission  effort 
for  the  US.  Additional  information  on  LibEcon  is  available  at  http://www.libecon2QQQ.org/. 

• Council  on  Library  and  Information  Resources  (CLIR)  initiative : CLIR  investigated  the  issues 
surrounding  network  statistics  primarily  from  an  online  database  data  perspective.  The  initial 
study,  conducted  during  1999  and  2000,  resulted  in  the  publication  of  a white  paper  entitled  White 
Paper  on  Electronic  Journal  Usage  Statistics.  It  is  the  understanding  of  the  study  team  that  the 
work  begun  through  this  effort  continues.  Additional  information  on  this  initiative  is  available  at 
http://www.clir.org. 

• Publisher  and  Libraries  Solution  Committee  (PALS):  This  recent  initiative,  operating  through  the 
auspices  of  the  Joint  Information  Systems  Committee  (JISC),  is  exploring  the  data  needs  of 
libraries  from  publisher  provided  online  usage  statistics.  A vendor-based  usage  statistics  working 
group  has  been  developed  to  explore  the  issues  involved  regarding  online  vendor  statistics  in  a 
more  in-depth  fashion.  This  group  met  in  June  2001  in  the  United  Kingdom  to  pursue  further 
network  statistics  needs  of  libraries  from  vendors.  Study  team  members  are  in  contact  with  this 
group.  Additional  information  on  this  initiative  is  available  at 
http://www.jisc.ac.Uk/curriss/collab/cb-pub/@wg-mem 

These  efforts  suggest  that:  (1)  There  is  great  interest  in  the  development  of  library  network  statistics  and 
performance  measures  across  a wide  range  of  interests,  perspectives,  and  stakeholder  groups;  (2)  The 
number  of  efforts  provide  some  idea  of  the  complexity  of  coordination,  cross-fertilization,  and  duplication 
of  effort  currently  in  progress;  and  (3)  It  is  possible  to  identify,  define,  and  institutionalize  a number  of 
network  statistics  and  performance  measures  and  attempt  to  blend  these  initiatives.  The  level  of  effort 
required  to  do  so,  however,  is  great  and  beyond  the  resources  of  the  study  team  - except  in  some  selective 
fashion. 


1.5.2  ARL  New  Measures  Initiative 

ARL  has  undertaken  a comprehensive  New  Measures  Initiative  through  which  it  attempts  to  capture 
research  library  activities  and  efforts  along  a number  of  outcomes-based  dimensions.  In  particular,  the 
initiative  focuses  on  the  need  to  articulate  library  outcomes  in  relation  to  the  communities  they  serve. 
Moreover,  the  New  Measures  initiative  stresses  the  importance  for  research  libraries  to  decide  what  their 
outcomes  should  be  and  how  to  measure  them  before  these  issues  are  decided  for  them.  Thus,  in  1994 
(Kyrillidou  & Crowe,  1998), 

[a]  new  strategic  objective  was  adopted  by  the  ARL  membership  to 
describe  and  measure  “the  performance  of  research  libraries  and  their 
contributions  to  teaching,  research,  scholarship  and  community  service.” 

This  action  ratified  new  directions  for  the  ARL  Statistics  and 
Measurement  Program  to  expand  beyond  measures  of ‘input’  (such  as 
collection  size,  number  of  staff,  expenditures,  etc.)  and  to  search  for  new 
kinds  of  measures  of  library  performance  and  impact. 

To  this  end,  ARL  later  took  on  the  New  Measures  Initiative,  which  is  comprised  of  the  E-Metrics, 
LibQUAL+™,  Research  Outcomes,  and  Educational  Outcomes  projects,  among  others. 
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The  idea  of  the  New  Measures  Initiative  is  to  identify  the  contribution  that  the  research  library  makes 
to  its  service  community  along  a number  of  fronts,  of  which  a select  few  are  described  below  (see 
http://www.arl.org/stats/newmeas/newmeas.html  for  additional  detail  on  the  New  Measures  Initiative): 

• LibQUAL+m : This  research  demonstration  project  seeks  to  define  and  measure  library  service 
quality  across  institutions  and  to  create  useful  quality-assessment  tools  for  local  planning.  The 
goals  of  LibQUAL+™  are  to  1)  establish  a library  service  quality  assessment  program  at  ARL;  2) 
Develop  web-based  tools  for  assessing  library  service  quality;  3)  Develop  mechanisms  and 
protocols  for  evaluating  libraries;  and  4)  Identify  best  practices  in  providing  library  service. 

• Educational  Outcomes : This  research  demonstration  project  focuses  on  the  role  of  the  academic 
library  in  the  support  of  overall  institutional  goals.  In  particular,  the  project  seeks  to  make  sense 
of  the  new  environment  research  libraries  (and  their  academic  institutions)  face  and  understand 
(1)  The  changing  expectations  faced  by  universities;  (2)  How  universities  are  responding  to  these 
new  expectations;  (3)  How  this  affects  the  library’s  mission;  (4)  How  libraries  can  develop  a 
strategy  to  be  a central  part  of  the  university’s  response;  and  (5)  How  the  Association  of  Research 
Libraries  can  assist  its  members  to  take  on  new  roles. 

• Research  Outcomes : The  research  outcomes  project  is  currently  underway  with  an  investigation 
being  conducted  by  Douglas  Jones  who  has  been  appointed  a Visiting  Program  Officer  for  the 
Association  of  Research  Libraries  to  identify  issues  and  explore  possible  measures  that 
demonstrate  the  role  of  the  library  in  support  of  the  university  research  mission. 

• Other  ‘new  measures’  project  focus  on  cost  studies  (i.e.,  technical  services,  interlibrary  loan,  and 
staff  allocation  cost  drivers)  in  ARL  libraries. 

The  E-Metrics  project,  of  which  this  study  is  a part,  seeks  to  understand  library  network  activities  and 
ways  in  which  to  capture,  report,  and  demonstrate  the  outcomes  of  those  services  and  resources. 
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2 FIELD-TESTING  METHODOLOGY  AND  SUMMARY  OF  FINDINGS 

While  the  proposed  statistics  and  measures  could  have  been  recommended  for  adoption  without  testing 
their  feasibility,  it  was  considered  prudent  to  subject  these  measures  to  some  form  of  testing.  Particularly, 
in  view  of  the  fact  that  many  of  the  proposed  statistics  and  measures  had  never  been  collected  in  a 
systematic  way,  we  needed  to  know  (1)  whether  the  statistics  and  measures  could  be  collected;  (2) 
whether  the  recommended  procedures  would  facilitate  data  collection;  (3)  the  estimated  time  and  efforts 
required  to  collect  data;  and  (4)  the  utility  of  statistics  given  the  amount  of  time  and  effort  to  collect  them, 
among  other  things.  Considerable  preparation  and  effort  by  participating  libraries  and  the  study  team 
went  into  the  field-testing  of  proposed  measures. 

This  section  describes  the  evolutionary  process  by  which  various  versions  of  statistics  and  measures, 
as  well  as  field-testing  methods  and  procedures,  were  refined.  It  also  presents  findings  related,  directly 
and  indirectly,  to  field-testing  results. 

2.1  Evolutionary  nature  of  statistics  for  networked  resources  and  services 

The  ARJL  E-Metrics  Project  represents  the  first  systematic  attempt  to  develop  statistics  and  measures 
to  describe  networked  information  resources  and  services  at  research  libraries  in  North  America.  It  also  is 
an  early  attempt  to  standardize  key  measures  and  definitions.  No  doubt  the  nature  and  focus  of  the 
statistics  will  evolve  as  the  research  library  environment,  information  technologies  in  particular,  changes 
over  time.  This  section  describes  the  process  in  which  statistics  and  measures  evolved  from  the  initial 
recommendation  to  the  adoption  for  the  field-testing  in  the  E-Metrics  context. 

The  measures  produced  from  the  project  are  intended  for  wider  adoption  by  ARL  libraries  but  may 
not  be  applicable  to  all  ARL  libraries  in  all  situations.  It  may  be  necessary  to  develop  additional,  more 
detailed  statistics  and  measures  at  the  individual  library  level  or  among  a group  of  libraries  sharing  similar 
interests. 


2.1.1  Initial  List 

Based  on  the  needs  assessment  study  conducted  in  Phase  I of  the  project,  the  study  team  issued  the 
first  preliminary  statistics  in  a memo  to  project  participants  dated  November  27,  2000  (see  Figure  2.1). 
The  list  of  statistics  also  appeared  in  the  December  2000  issue  of  the  ARL  Newsletter 
(http://www.arl.org/newsltr/213/index.html). 
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Figure  2.1  Preliminary  Statistics  and  Measures  (E-Metrics  version  1.0) 

Statistics 

□ Electronic  Resources  and  Services 

• Number  of  electronic  full-text  journals  (hosted  by  library) 

• Number  of  librarians  providing  electronic  reference 

• Virtual  visits  to  networked  library  resources 

• Electronic  reference  transactions 

• Number  of  public  access  workstations 

□ Electronic  Databases 

• Number  of  electronic  full-text  journals  (through  subscription) 

• Logins  (sessions) 

• Queries  (searches) 

• Items  examined  (viewed,  downloaded,  emailed,  printed) 

• Tum-aways  (requests  exceeded  simultaneous  user  limit) 

• Total  user  connection  time  to  vendor  databases 

a Instruction 

• Number  of  people  participating  in  user  instruction  on  electronic  resources 

□ Cost  of  electronic  databases  and  services 

• Cost  of  electronic  database  subscriptions 

• Cost  of  internal  digital  collection  construction 

•  Cost  per  items  examined  (subscribed  databases) 

Measures 

• Percentage  of  electronic  reference  transactions  of  total  reference 

• Percentage  of  electronic  materials  use  of  total  library  materials  use 

• Percentage  of  remote  library  visits  of  all  library  visits 

• Ratio  of  public  access  workstations  to  university  population  (number  of  faculty,  staff,  and 
students) 


The  list  is  divided  into  two  major  sections:  one  containing  statistics  and  the  other  performance 
measures  that  can  be  derived  using  combinations  of  individual  statistics.  The  statistics  section  is  further 
divided  into  four  main  categories  of  resources/services,  electronic  databases,  instruction,  and  cost.  The 
study  team  received  comments  from  five  libraries,  many  of  them  asking  for  clarifications  on  the 
definitions. 


2.1.2  Revised  List 

Based  on  the  comments,  the  study  team  modified  and  released  a revised  list  with  brief  definitions. 
The  revised  list  was  first  introduced  during  the  CNI  Fall  2000  Task  Force  Meeting  on  December  7,  2000, 
in  San  Antonio,  Texas,  and  later  sent  to  project  participants  on  December  14,  2000,  as  a survey. 
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Figure  2.2  Revised  Statistics  and  Measures  (E-Metrics  version  1.1) 

Statistics 

□ Patron  Accessible  Resources 

• Number  of  electronic  full-text  journals  (hosted  by  library) 

• Number  of  electronic  full-text  periodicals  (through  institutional  subscription) 

• Number  of  electronic  full-text  periodicals  (through  consortia  and  other  arrangements) 

• Number  of  electronic  reference  databases  (through  institutional  subscription) 

• Number  of  electronic  reference  databases  (through  consortia  and  other  arrangements) 

• Number  of  electronic  books 

• Number  of  staff  providing  electronic  reference 

• Number  of  public  access  workstations 

□ Use  of  Electronic  Resources  and  Services 

• Number  of  logins  (sessions)  to  networked  library  resources 

• Electronic  reference  transactions 

• Number  of  logins  (sessions)  to  electronic  databases 

• Queries  (searches) 

• Total  connection  time  to  electronic  databases 

• Items  examined  (viewed,  downloaded,  emailed,  printed)  to  electronic  databases 

• Tum-aways  (requests  exceeded  simultaneous  user  limit) 

□ Instruction 

• Number  of  people  participating  in  user  instruction  on  electronic  resources 

□ Cost  of  Electronic  Databases  and  Services 

• Cost  of  electronic  files  (one-time/monographic  purchase) 

• Cost  of  electronic  full-text  periodicals  subscriptions 

• Cost  of  electronic  reference  database  subscriptions 

• Library  contribution  to  consortia  for  electronic  databases 

•  Cost  of  internal  digital  collection  construction 

Measures 

• Percentage  of  electronic  reference  transactions  of  total  reference 

• Percentage  of  electronic  materials  use  of  total  library  materials  use 

• Percentage  of  remote  library  visits  of  all  library  visits 

• Percentage  of  electronic  titles  to  all  periodicals 

• Percentage  of  electronic  books  to  all  monographs 

• Ratio  of  public  access  workstations  to  university  population 

•  Cost  per  items  examined  in  individually  subscribed  databases 

• (institutional)  means  through  institutional  subscription  and  (consortia)  means  through  consortia  subscription. 

The  key  change  from  the  initial  list  to  the  revised  list  was  clearer  categorization  of  statistics: 
infrastructure  (resources  that  can  be  accessed  by  users),  use  of  the  infrastructure,  and  the  cost  to  support  it. 
Another  change  was  the  inclusion  of  the  ARL  supplementary  cost  statistics  with  the  intention  of  avoiding 
duplicated  efforts.  As  a result,  the  number  of  statistics  increased  from  15  in  the  initial  list  to  21  in  the 
revised  list,  and  from  4 performance  measures  to  7. 

In  the  survey  of  project  participants,  the  study  team  asked  respondents  to  rate  the  usefulness  of  the 
statistics  and  measures  on  a 5-point  scale  (5  being  the  most  useful)  and  make  comments  and  suggestions 
for  each  proposed  statistic  and  measure.  The  usefulness  rating  was  intended  to  give  the  study  team  a 
quick,  quantitative  measure  of  any  potential  problem  with  statistics  and  measures. 
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The  study  team  received  a total  of  12  responses  from  10  institutions.  One  response  came  in  the  form 
of  an  overall  comment  and  did  not  have  the  usefulness  ratings  the  survey  sought.  Overall,  respondents 
gave  favorable  marks  to  the  usefulness  of  the  proposed  statistics  and  measures.  The  average  rating  from 
1 1 responses  was  4.2  out  of  5.  There  were  five  responses  whose  average  was  4.5  and  above,  with  a high 
of  4.9.  There  were  only  two  responses  whose  average  was  below  4.0,  with  a low  of  3.3. 

Category-wise,  respondents  felt  most  comfortable  with  statistics  related  to  cost  aspects  of  electronic 
resources  and  services  and  least  comfortable  with  the  proposed  performance  measures.  Table  2.1  shows 
the  average  usefulness  score  for  5 categories  of  revised  statistics  and  measures 

Table  2.1  Usefulness  rating  for  revised  safeties  and  measures  by  category 


Category 

Usefulness 

(Average) 

Patron  accessible  resource 

4.3 

Use  of  electronic  resources  and  services 

4.2 

User  instruction 

4.0 

Cost  of  electronic  services  and  resources 

4.6 

Performance  measures 

3.9 

Among  the  statistics  associated  with  patron  accessible  resources,  two  statistics — number  of  staff 
providing  electronic  reference  and  number  of  public  access  workstations — were  identified  as  more 
problematic  or  less  useful.  The  usefulness  rating  was  3.5  for  both  statistics.  Among  the  use  statistics, 
total  connection  time  was  the  only  statistic  that  garnered  a usefulness  score  under  4.0  with  a score  of  3.6. 
Cost  of  internal  digital  collection  construction  was  seen  as  the  least  useful  among  cost  statistics  with  a 
score  of  3.9.  In  the  performance  measures  category,  respondents  identified  percentage  of  electronic 
materials  use  of  total  library  materials  use  (3.2),  ratio  of  public  access  workstations  to  university 
population  (3.7),  and  cost  per  items  examined  in  individually  subscribed  databases  (3.9)  less  useful  than 
other  performance  measures. 

The  reasons  for  low  usefulness  scores  were  twofold.  It  appears  that  usefulness  was  interpreted  as  both 
practical  (that  the  statistic  can  be  collected)  and  worthwhile  (it  tells  something  or  answers  a question). 
There  was  clearly  a sense  that  many  of  the  statistics  and  measures,  while  potentially  useful,  would  be 
fairly  difficult  to  keep  track  of. 

During  the  project  meeting  in  conjunction  with  the  ALA  Mid-Winter  Meeting  on  January  12,  2001,  in 
Washington,  22  participants  from  17  institutions  were  asked  to  choose  the  top  five  statistics  they  would 
like  to  have.  The  rankings  for  the  most  part  agreed  with  the  usefulness  scores  described  above. 

2.1.3  Field-testing  List 

Based  on  participants’  input,  the  study  team  drafted  a set  of  statistics  and  measures  for  field-testing  as 
listed  in  Figure  2.3. 
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Figure  2.3  Statistics  and  Measures  Field  Tested  (E-Metrics  version  2.0) 

Statistics 

□ Patron  Accessible  Resources 

• Number  of  electronic  full-text  journals  (institutional) 

• Number  of  electronic  full-text  journals  (consortia) 

• Number  of  electronic  reference  sources  (institutional) 

• Number  of  electronic  reference  sources  (consortia) 

• Number  of  electronic  books  (institutional) 

• Number  of  electronic  books  (consortia) 

□ Use  of  Electronic  Resources  and  Services 

• Number  of  electronic  reference  transactions 

• Number  of  logins  (sessions)  to  electronic  databases 

• Number  of  queries  (searches)  in  electronic  databases 

• Items  examined  in  electronic  databases 

□ Cost  of  Electronic  Databases  and  Services 

• Cost  of  electronic  full-text  journals 

• Cost  of  electronic  reference  sources 

• Cost  of  electronic  books 

• Library  expenditures  for  bib.  utilities,  networks,  and  consortia 

• External  expenditures  for  bib.  utilities,  networks,  and  consortia 

□ Local  Digital  Collection  Statistics 

• Size  of  library  digital  collection 

• Use  of  library  digital  collection 

•  Cost  of  digital  collection  construction  and  management 

Measures 

• Percentage  of  electronic  reference  transactions  of  total  reference 

• Percentage  of  electronic  materials  use  of  total  library  materials  use 

• Percentage  of  virtual  library  visits  of  all  library  visits 

•  Percentage  of  electronic  books  to  all  monographs 


The  field-testing  list  contains  a separate  category  for  local  digital  collections,  which  added  two  more 
statistics  (size  of  the  local  digital  collection  and  use  of  the  collection).  In  the  patron  accessible  resource 
category,  three  main  resource  types  are  identified  (full-text  journals,  reference  databases,  and  electronic 
books)  and  each  divided  into  institutional  and  consortia  subscriptions.  Two  statistics  were  dropped  in  the 
resource  category — number  of  public  workstations  and  number  of  librarians  providing  electronic 
reference.  In  the  usage  statistics  category,  the  number  of  statistics  was  reduced  from  7 to  4.  Overall,  the 
number  of  statistics  and  measures  was  reduced  from  28  in  the  E-Metrics  version  1.1  (revised  list)  to  22  in 
version  2.0  (field  test  list). 

Table  B.  1 (in  Appendix  B)  lists  the  statistics  and  measures  included  in  each  iteration. 

2.1.4  Summary 

Through  the  iterative  process  the  study  team  was  able  to  propose  a set  of  statistics  and  measures  for 
field-testing.  In  each  iteration,  inputs  from  project  participants  were  sought  and  used  as  guidelines  for 
modification.  The  process  was  a combined  effort  between  the  members  and  the  study  team.  Field-testing 
provided  another  opportunity  to  zero  in  on  the  feasibility  of  the  statistics  and  measures.  The  results  of  the 
field-testing  and  subsequent  revision  of  statistics  and  measures  are  described  in  Section  2.2. 
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The  categories  used  in  classifying  statistics  and  the  statistics  themselves  show  that  the  current  effort  is 
still  focused  on  the  library  input-output  framework  of  performance  evaluation.  Most  noticeable  is  the 
attempt  to  construct  statistics  and  measures  of  electronic  resources  and  services  that  are  analogous  to  their 
print  counterparts.  The  statistics  and  measures  proposed  provide  for  research  libraries  means  of 
accounting  for  resources  and  services  delivered  through  electronic  channels,  and  thus  a better  view  of  and 
justification  for  the  changing  face  of  libraries. 

The  current  effort  extends  the  input-output  models  of  measurement.  However,  these  statistics  and 
measures  are  essential  for  future  work  and  for  developing  new  models  or  frameworks  to  move  toward  a 
more  user-  and  outcome-oriented  approach  (see  Section  1.4  for  further  discussion).  Moreover,  these  are 
the  statistics  that  people  felt  comfortable  with,  as  evidenced  in  the  high  usefulness  ratings  given  to  these 
statistics  and  measures.  We  described  the  evolutionary  process  that  took  place  in  developing  statistics  and 
measures  for  electronic  resources  and  services  in  the  context  of  the  ARL  E-Metrics  Project.  We 
acknowledge  that  there  are  forces  - technology,  market,  government  regulations,  and  so  on  - that  affect  the 
development  of  such  statistics  and  measures.  We  encourage  research  libraries  to  continually  engage  in 
validating  statistics  and  measures  of  these  realities  and  updating  them  to  better  reflect  the  activities  and 
accomplishments  of  research  libraries  as  a whole  or  of  individual  institutions. 
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2.2  Field-testing  Logistics 

Having  discussed  the  developments  that  led  to  the  selection  of  statistics  and  measures  for  the  field- 
testing  in  Section  2. 1,  we  need  to  explain  some  of  the  logistics  involved. 

The  ARL  E-Metrics  participants  had  several  chances  to  reflect  upon  and  comment  on  several 
iterations  of  the  proposed  set  of  statistics  and  measures  that  the  study  team  drafted.  Also,  during  the 
project  meeting  at  the  ALA  Mid- Winter  meeting  in  Washington,  DC  (January  12,  2001),  attendees  had  an 
opportunity  to  rank  the  top  five  statistics  among  each  category  (for  example,  statistics  related  to  patron 
accessible  resources)  based  on  perceived  usefulness.  The  idea  was  to  weed  out  statistics  and  measures 
that  were  not  worthwhile  to  collect  (through  averaging  usefulness  rankings,  which  is  a way  of  consensus 
building),  and  thus  narrow  the  number  of  measures  to  be  tested.  As  a result,  the  total  number  of  measures 
decreased  from  28  to  22. 

Regarding  the  field-testing  logistics,  the  study  team  presented  the  following  three  options: 

• All  or  nothing  where  a small  set  of  libraries  field  test  all  of  the  proposed  statistics; 

• Area  coverage  where  a small  set  of  libraries  collect  the  same  category  of  statistics  (for  example, 
cost-related  measures  only);  and 

• Pick  and  choose  where  all  libraries  choose  from  a full  set  of  statistics  what  they  would  like  to 
field  test. 

The  relative  advantages  and  disadvantages  associated  with  each  option  are  summarized  in  Table  2.2. 


Table  2.2  Comparison  of  Field-testing  Options 


Options 

Advantages 

Disadvantages 

All  or  Nothing 

Easy  to  administer  and  gather  results 

Great  burden  on  libraries  to  use  all 
statistics 

Area  Coverage 

Lesser  burden  on  all  libraries 

Smaller  set  of  results,  hard  to  adopt  by 
libraries  not  in  the  field  test 

Pick  and  Choose 

Large  set  of  results,  easy  for  libraries 
to  adopt 

Hard  to  administer  and  analyze  results 

All  of  these  options  were  discussed  during  the  project  meeting  at  the  ALA  Mid-Winter  meeting  and  it 
was  agreed  that  the  participation  should  be  broad  enough  to  warrant  meaningful  results.  It  was  also 
agreed  that  if  a library  decided  to  field  test,  it  would  field  test  all  of  the  statistics  and  measures,  so  that  the 
results  would  be  more  reliable  and  consistent.  The  study  team  made  one  exception  to  this:  libraries  could 
choose  not  to  field  test  statistics  related  to  digital  library  collection  since  all  libraries  were  not  heavily 
invested  (or  involved,  developed)  in  digital  collection  building. 

After  the  meeting,  the  study  team  drafted  and  sent  a final  set  of  field-testing  statistics  and  measures 
with  brief  definitions  to  the  participants  and  asked  them  to  indicate  first  whether  they  would  like  to 
participate  in  the  field-testing  and  second  what  level  of  participation  they  would  like  to  have  (all  statistics 
including  digital  collection  statistics  or  all  statistics  excluding  digital  collection  statistics). 

A total  of  16  libraries  decided  to  participate  in  the  field-testing.  Among  them,  twelve  (12)  libraries 
were  the  ones  who  were  going  to  test  all  proposed  statistics  and  measures  and  the  other  four  (4),  all  but 
the  library  digitization  statistics  (refer  to  Table  2.3).  A liaison  at  each  institution  was  identified  as  a 
primary  contact  during  the  field-testing. 
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Table  2.3  Field-testing  Libraries 


Participants  (All  proposed  statistics  and 
measures) 

Participants  (all  but  library  digitization 
statistics) 

Alberta,  Arizona  State,  Chicago,  Cornell,  Illinois  at 
Chicago,  Maryland,  Massachusetts,  Nebraska, 
Pittsburgh,  Texas  (Austin),  Virginia  Tech,  and 
NYPL 

Auburn,  Manitoba,  Western  Ontario,  Yale 

While  the  field-testing  libraries  and  liaisons  were  being  identified,  the  study  team  worked  on  the  field- 
testing  instruction  manual.  Writing  the  instructions  was  not  an  easy  process  because  we  needed  detailed 
definitions,  examples,  and  data  collection  procedures.  The  field-testing  manual  was  prepared  in 
consultation  with  many  people,  within  and  outside  the  study  team.  While  the  set  of  instructions  was  not 
intended  as  a complete  and  fully  developed  procedural  guide,  it  evolved  into  a substantial  document,  with 
all  the  forms  required  for  data  collection  and  reporting  purposes.  The  finished  instruction  manual  was 
sent  out  to  field-testing  libraries  on  March  26,  2001. 

We  included  in  the  field-testing  instructions  a self-evaluation  form  for  each  statistic  and  measure  in 
order  to  understand  how  the  local  testing  was  carried  out,  what  each  library  learned  during  the  process, 
and  what  might  be  the  likely  issues  if  these  statistics  were  to  be  collected  on  a regular  basis.  The  form 
asked  the  following  questions: 

1 . What  was  the  easiest  aspect  of  collecting  the  data  for  this  statistic/measure? 

2.  What  was  the  most  difficult  aspect  of  collecting  the  data  for  this  statistic/measure? 

3.  Do  you  have  any  suggestions  and  recommendations  that  would  improve  the  data  collection 
process? 

4.  Approximately  how  many  staff  hours  were  necessary  to  produce  this  statistic/measure? 

5.  To  what  degree  is  this  statistic/measure  worth  collecting  given  the  amount  of  time  and  effort 
required  to  collect  it? 

During  the  field-testing,  the  study  team  received  numerous  questions  from  library  liaisons  regarding 
various  aspects  of  field-testing  (e.g.,  definitions,  procedures  and  use  of  forms)  and  provided  answers  and 
guidelines  according  to  the  field-testing  instructions. 

2.2.1  Site  Visits 

During  the  week  of  April  23,  2001,  Wonsik  ’’Jeff'  Shim  conducted  site  visits  at  three  field-testing 
libraries — University  of  Chicago,  University  of  Illinois  at  Chicago,  and  University  of  Manitoba.  Jeff  met 
with  library  staff  members  who  were  involved  in  the  field-testing,  answered  questions,  and  collected 
information  related  to  field-testing  procedures. 

Both  one-on-one  interviews  with  the  site  liaison  and  group  interviews  with  key  informants  were 
conducted.  The  visits  provided  very  useful  information  about  the  detailed  processes  that  these  libraries 
adopted  to  complete  the  field-testing,  which  otherwise  would  not  have  been  known. 

2.3  Summary  of  Results 

Three  kinds  of  data  sources  informed  the  results  summarized  in  this  section: 
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• Reported  statistics  and  measures 

• Self-evaluation  comments 

• Site  visits 

Some  of  the  findings  made  their  way  into  the  final  data  collection  manual  included  in  this  report  and  will 
not  be  discussed  here.  We  instead  focus  on  the  overall  management  of  data  collection  efforts  and  lessons 
learned  during  field-testing. 

Overall,  14  out  of  16  libraries  were  able  to  compile  and  report  data.  Table  2.4  shows  the  level  of 
participation,  at  least  to  the  extent  that  numbers  were  reported  by  each  library.  The  names  of  the  libraries 
are  suppressed  as  the  information  is  not  integral  for  reporting  purposes.  Note  that  statistics  U2  (number  of 
session  in  licensed  databases),  U3  (number  of  queries),  and  U4  (number  of  items  requested)  were  not 
included  in  the  library  field-  testing  as  they  were  tested  separately  with  database  vendors  (P2  was  not 
tested  either  as  it  requires  U4  for  calculation). 

The  table  shows  that,  even  excluding  highly  experimental  digital  collection  statistics  (D1-D3),  many 
libraries  could  not  report  a number  of  data  elements.  For  example,  Library-04  is  missing  three  cost 
statistics  (Cl,  C4,  and  C5),  two  digital  collection  statistics  (D1  and  D2),  and  one  performance  measure 
(P3).  On  the  other  hand,  Library  1 1 was  able  to  report  all  but  one  cost  statistic  (C5). 

It  appears  that  although  libraries  were  able  to  report  statistics  related  to  electronic  resources,  they 
seem  to  have  had  more  difficulty  collecting  statistics  related  to  expenditures  (C1-C5,  in  particular  C5). 
Moreover,  just  because  libraries  were  able  to  report  certain  statistics,  the  data  collection  was  not 
necessarily  complete  and  easy.  Even  a cursory  reading  of  the  comments  in  the  evaluation  forms  makes  us 
believe  that,  in  some  cases,  numbers  were  generated  at  a cost  of  many  staff  hours  and  development  of 
local  procedures  to  collect  them. 


Table  2.  4 Field-testing  Participation  Results 
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♦ : data  reported 

n/a:  unable  to  report  or  not  applicable  (libraries  who  were  not  testing  digital  collection  statistics) 
x : no  submission 


A majority  of  libraries  reported  electronic  reference  transactions  (Ul).  In  the  cost  statistics  (Cl 
through  C5),  C2  (cost  of  electronic  reference  sources)  was  most  readily  available  by  field-testing  libraries 
whereas  C5  (external  expenditures  for  bibliographic  utilities,  networks  and  consortia)  was  reported  by 
only  a handful.  The  majority  of  missing  performance  measures  data  (PI,  P3,  and  P4)  is  due  to  missing 
base  statistics  data  elements  (R3,  Ul,  U2)  required  to  calculate  performance  measures. 
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Table  2.5  shows  another  picture  of  the  data  collection  efforts.  It  shows  the  number  of  staff  hours 
spent  to  collect  each  category  of  data  elements.  While  the  field-testing  instructions  required  that  field- 
testing  libraries  keep  track  of  staff  hours  devoted  to  preparing  and  carrying  out  data  collection  activities, 
we  believe  that  the  reported  hours  are  an  underestimation.  We  estimate  that  more  stringent  time  keeping 
could  have  resulted  in  increased  reported  staff  hours. 

Looking  at  the  total  number  of  hours  spent,  there  is  a wide  range  from  only  7 hours  to  164  hours, 
which  is  equivalent  to  one  staff  member  spending  4.5  weeks  (based  on  37.5  hour  week)  for  field-testing. 
It  is  difficult  to  tell,  just  by  looking  at  the  number  of  hours  spent,  which  libraries  were  more  efficient  in 
executing  field-testing. 


Table  2.5  Approximate  time  taken  to  collect  data  (in  hours) 


Resource 

Cost 

Use 

Digital 

Performance 

Total 

Library-01 

12.00 

n/a 

2.00 

8.00 

n/a 

22.00 

Library-03 

15.50 

; 13.00 

0.50 

16.00 

n/a 

45.00 

Library-04 

32.50 

32.00 

2.00 

1.00 

3.50 

71.00 

Library-05 

13.00 

6.25 

2.00 

7.50 

3.00 

31.75 

Library-06 

31.00 

27.50 

1.00 

n/a 

12.00 

71.50 

Library-07 

32.00 

n/a 

n/a 

n/a 

n/a 

32.00 

Library-08 

1.00 

0.25 

8.00 

n/a 

n/a 

9.25 

Library-09 

1.00 

n/a 

2.00 

20.00 

0.50 

23.50 

Library- 10 

39.00 

1.00 

2.50 

2.50 

14.00 

59.00 

Library- 1 1 

80.00 

2.00 

8.00 

70.00 

4.00 

164.00 

Library- 12 

6.00 

1.00 

n/a 

n/a 

n/a 

7.00 

Library- 13 

14.00 

7.00 

n/a 

n/a 

n/a 

21.00 

Library  2 and  14  did  not  report  the  data, 
n/a:  not  available  or  not  applicable 


Library- 1 1,  which  was  by  far  the  most  involved  field  test  library  based  on  the  number  of  hours  spent, 
not  surprisingly  reported  the  most  complete  data  (refer  to  Table  2.5).  It  was  clear  from  the  reported  data 
and  the  comments  that  the  library  had  internal  data  collection  resources  (such  as  an  electronic  resource 
management  database)  not  available  in  many  other  field-testing  libraries.  Nonetheless,  it  tried  to  conform 
to  the  field-testing  definitions  and  procedures,  which  resulted  in  high  staff  hours  spent. 

On  the  other  hand,  Library-12  also  was  able  to  report  more  statistics  than  average  participants  while 
spending  minimal  hours  (7  hours),  largely  because  it  used  rough  numbers  that  it  already  had  without 
investing  extra  efforts  to  refine  those  and  trying  to  conform  to  the  field-testing  instructions. 

Simple  comparison  of  staff  hours  spent  does  not  indicate  any  conclusive  relationship  between  the 
completeness  of  data  and  the  invested  efforts.  Library  4,  spent  at  least  7 1 hours  but  missed  quite  a few 
statistics.  On  the  other  hand,  Library  13  spent  21  hours  and,  predictably,  was  able  to  report  only  a few 
statistics. 

One  clear  conclusion  from  the  results  is  that  there  is  a varying  level  of  effort.  But  it  is  not  certain 
whether  the  levels  of  effort  at  these  libraries  will  remain  the  same  when  these  statistics  become  regularly 
collected  and  reported.  Better  internal  systems  that  support  this  kind  of  data  collection  combined  with 
more  routine  and  consistent  data  collection  procedures  will  certainly  improve  the  efficiency  of  data 
collection  operations. 

Many  libraries  commented  that  a good  portion  of  their  time  was  spent  trying  to  understand  the  field- 
testing  instructions  and  establishing  local  procedures  and  organizational  arrangements.  For  example, 
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Library-01  reported  “3  FTE  staff  spent  4 hours  compiling  the  data  for  measures  R1-R6.  Initial  8 hours  to 
develop  the  local  in-house  electronic  resources  spreadsheet.” 

A number  of  libraries  also  commented  that  they  spent  many  hours,  largely  because  this  was  the  first 
time  they  collected  these  statistics,  but  they  felt  that  ongoing  data  collection  would  improve  in  terms  of 
efficiency. 

Field-testing  was  time  consuming  partly  due  to  the  artificial  requirements  imbedded  in  the  field- 
testing.  For  example,  we  asked  more  detailed  information  than  would  be  necessary  in  an  ongoing  regular 
data  collection  to  make  sure  that  there  was  consistency  in  reporting  data.  Without  asking  for  detailed 
data,  we  would  have  had  no  way  of  knowing,  for  example,  whether  what  one  library  treated  as  a full-text 
journal  was  treated  similarly  by  other  libraries.  This  created  an  extra  burden  on  the  libraries  and  may 
have  contributed  to  some  of  the  missing  data.  We  believe  that  with  more  relaxed  requirements,  we  could 
have  avoided  situations  in  which  one  library  spent  “4  man-hours  to  verify  that  we  could  not  do  this” 
while  another  devoted  “8  hours  until  the  effort  was  abandoned.”  However,  more  lax  requirements  would 
have  exacerbated  the  already  significant  problem  of  inaccurate,  inconsistent,  and  unreliable  statistics. 

The  following  points  summarize  what  was  learned  from  analyzing  the  time  taken  to  carry  out  field- 
testing  and  library  comments: 

• It  is  likely  that  libraries  will  spend  varying  amounts  of  time,  effort,  and  resources  to  conduct 
ongoing  collection  of  statistics  and  measures  related  to  electronic  resources  and  services.  The 
degree  of  effort  depends  on  the  library’s  capability  (resources)  and  interest  in  data  collection  and 
use. 

• With  investment  in  internal  information  systems  and  establishing  ongoing  local  procedures,  the 
effort  to  collect  these  statistics  can  be  decreased  significantly.  However,  given  the  fact  that 
current  metrics  will  evolve  over  time  and  that  individual  libraries  will  want  to  collect  local 
specific  data,  we  believe  that  libraries  will  need  to  commit  a significant  amount  of  time  and  effort 
to  plan  for  and  collect  data  regarding  electronic  resources  and  services. 

• Standardized  definitions  and  procedures  will  improve  data  collection  and  lead  to  consistent 
reporting  of  these  statistics. 

2.3.1  Usefulness  of  statistics  and  measures  gathered 

In  the  self-evaluation  form,  we  asked  libraries  to  assess  the  usefulness/value  of  each  statistic  relative 
to  the  amount  of  time  and  effort  spent.  The  first  thing  we  noticed  was  that  there  were  not  many  outright 
rejections  of  statistics  and  measures  due  to  a perceived  lack  of  usefulness.  As  for  the  reasons  some 
statistics  and  measures  were  considered  potentially  useful,  we  did  not  find  a wide  range  of  responses. 
Typical  examples  of  use  were  trend  plotting,  benchmarking,  and  reporting.  Perhaps  the  question  itself 
was  not  specific  enough:  we  did  not  ask  what  kinds  of  questions  could  be  answered  by  having  these 
questions,  only  the  degree  to  which  a statistic  was  useful.  Another  explanation  for  the  lack  of  specificity 
may  have  been  that  the  statistics  and  measures  tested  are  more  or  less  gross  figures  and  by  themselves 
may  not  directly  relate  to  specific  decision-making  situations.  One  could  also  point  to  the  fact  that  the 
comments  and  evaluations  reflect  the  views  of  people  who  replied  (librarians)  and  may  not  encompass 
others’  views  (such  as  library  directors  or  university  administrators).  Overall,  it  appears  that  libraries  saw 
these  measures  as  good  things  to  have  in  the  absence  of  more  detailed  data. 
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Table  2.6  contains  verbatim  statements  found  in  the  comments  to  Question  5.  It  shows  a range  of 
reasons  why  a statistic  or  a measure  may  or  may  not  be  considered  useful.  It  also  suggests  that  in  some 
cases,  a change  in  the  definitions  and  procedures  need  to  be  made. 


Table  2.  6 Examples  of  usefulness  statements 


Useful 

Not  Useful 

Other 

• Useful  for  benchmarking.  (Rl) 

• This  is  a very  useful  stat  to  keep  as 
we  anticipate  the  % of  e-reference  to 
total  reference  will  be  shifting.  (Ul) 

• Essential  for  fiscal  accounting  and 
reporting.  (C1-C3) 

• A best  buy  in  terms  of  benefit:cost. 
(C3) 

• Very  valuable,  as  it  captures  a 
significant  expenditure  made  on 
behalf  of  the  library.  (C5) 

• It  is  important  and  often  asked  for. 
(D2) 

• We  were  very  interested  in 
determining  personnel  costs  and 
found  the  data  gathering  worth  the 
effort.  (D3) 

• Very  useful.  The  stat  will  enable  us 
to  establish  trends  for  our  service 
areas  that  are  useful  for  planning 
purposes.  (PI) 

• Since  ebooks  are  only  recently 
becoming  more  prevalent,  it  will 
become  an  increasingly  important 
measure,  no  doubt.  (P4) 

• We  have  an  electronic  journal 
database,  so  this  was  actually 
easy  to  get.  I am  not  sure  why 
it  is  important.  (R2) 

• Not  very  useful,  though  not 
difficult.  (R4) 

• It  is  of  doubtful  value.  (Ul) 

• I don’t  know.  (C3) 

• It  seems  not  worth  collecting. 
As  defined  here  it  is  too 
imprecise  and  it  seems  to  be  a 
very  small  cost  compared  to 
the  other  ones.  (C4) 

• Not  worth  it.  What  we 
submitted  here  is  a partial 
answer,  and  an  incorrect  one 
at  that.  (D3) 

• For  us,  it  isn’t  worth 
the  time  if  we  must 
separate  institution 
subscriptions  from 
consortia  and  other. 
(Rl) 

• We  do  need  to 
simplify  this  statistics. 
(Cl) 

• The  overall  data  is 
worth  collecting.  The 
time  needed  to  prorate 
expenditures  is  cost- 
prohibitive.  (Cl) 

• Maybe  of  local  interest 
only.  Less  useful  for 
inter-institutional 
benchmarking.  (D2) 

* Notations  in  the  parentheses  refer  to  the  statistic  or  measure  to  which  the  comment  was  made. 


Based  on  the  comments  and  analysis  of  reported  data,  we  can  place  different  statistics  and  measures  in 
a 2 by  2 matrix,  according  to  their  usefulness  and  practicality  of  data  collection,  to  decide  whether  we 
need  to  include,  exclude,  or  make  changes  to  them.  Figure  2.4  shows  possible  actions  to  the  statistics  and 
measures  based  on  two  criteria  being  used. 


ARL  E-Metrics  Phase  II  Report 


38 


October  200 1 


183 


Y, 


Measures  and  Statistics  for  Research  Library  Networked  Services 


Figure  2.4  Actions  to  statistics  and  measures  based  on  usefulness  and  practicality 


Practicality 

2 

o 

Simplify  /Clarify 

Throw  Away 

WJ 

Keep 

Not  worth  it 

Yes 

No 

Usefulness/Value 

Again  we  emphasize  that  there  was  no  statistic  or  measure  tested  that  was  rejected  by  the  majority  of 
field-testing  librarians.  There  were  cases  where  changes  in  the  procedures  were  necessary  to  make  the 
data  collection  more  feasible  and  efficient.  As  a result,  most  of  the  tested  statistics  and  measures  gravitate 
toward  Keep  or  Simplify  /Clarify  quadrants. 


Table  2.7  shows  the  lists  of  statistics  and  measures  before  and  after  the  field-testing  with  brief 
explanations.  A revised  data  collection  manual  is  provided  in  this  report  using  the  new  set  of  statistics 
and  measures. 


Table  2.7  Changes  in  the  make-up  of  statistics  and  measures 


Field  Tested  Statistics  and 
Measures 

Statistics  and  Measures  After 
Field-testing 

Changes 

R1  Number  of  electronic  full-text 
journals  (through  institutional 
subscription) 

RI  Number  of  electronic  full-text 
journals 

Eliminated 

institutional/consortium 
subscription  distinction. 

R2  Number  of  electronic  full-text 
journals  (through  consortia  and 
other  arrangements) 

R3  Number  of  electronic  reference 
sources  (through  institutional 
subscription) 

R2  Number  of  electronic  reference 
sources 

R4  Number  of  electronic  reference 
sources  (through  consortia  and 
other  arrangements) 

R5  Number  of  electronic  books 
(through  institutional 
subscription) 

R3  Number  of  electronic  books 

R6  Number  of  electronic  books 
(through  consortia  and  other 
arrangements) 

U1  Number  of  electronic  reference 
transactions 

UI  Number  of  electronic  reference 
transactions 

Changed  the  unit  from 
questions  to  transactions 

U2  Number  of  logins  (sessions)  to 
electronic  databases 

U2  Number  of  logins  (sessions)  to 
electronic  databases 

Field-tested  separately. 
No  change 

U3  Number  of  queries  (searches)  in 
electronic  databases 

U3  Number  of  queries  (searches)  in 
electronic  databases 
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Field  Tested  Statistics  and 
Measures 

Statistics  and  Measures  After 
Field-testing 

Changes 

U4  Items  examined  in  electronic 
databases 

U4  Items  examined  in  electronic 
databases 

U5  Number  of  virtual  visits  to  the 
networked  library  resources 

Newly  created  to 
calculate  new  P2.  Was 
not  accounted  for 
previously. 

Cl  Cost  of  electronic  full-text  journals 

Cl  Cost  of  electronic  full-text 
journals 

Relaxed  the  expenditure 
prorating. 

C2  Cost  of  electronic  reference 
sources 

C2  Cost  of  electronic  reference 
sources 

C3  Cost  of  electronic  books 

C3  Cost  of  electronic  books 

C4  Library  expenditures  for 

bibliographic  utilities,  networks, 
and  consortia 

C4  Library  expenditures  for 

bibliographic  utilities,  networks, 
and  consortia 

No  change 

C5  External  expenditures  for 

bibliographic  utilities,  networks, 
and  consortia 

C5  External  expenditures  for 

bibliographic  utilities,  networks, 
and  consortia 

No  change 

DI  Size  of  library  digital  collection 

Dl  Size  of  library  digital  collection 

Emphasized  local  use  of 
data  and  de-emphasized 
cross  comparison 

D2  Use  of  library  digital  collection 

D2  Use  of  library  digital  collection 

D3  Cost  of  digital  collection 

construction  and  management 

D3  Cost  of  digital  collection 

construction  and  management 

PI  Percentage  of  electronic  reference 
transactions  of  total  reference 

PI  Percentage  of  electronic  reference 
transactions  of  total  reference 

No  change  ->  See  U 1 

P2  Percentage  of  electronic  materials 
use  of  total  library  materials  use 

Dropped  because  it  has 
too  many  components 
that  are  not  defined 
elsewhere. 

P3  Percentage  of  remote  library  visits 
of  all  library  visits 

P2  Percentage  of  virtual  library 
visits  of  all  library  visits 

No  change 

P4  Percentage  of  electronic  books  to 
all  monographs 

P3  Percentage  of  electronic  books  to 
all  monographs 

No  change 

2.3.2  Other  challenges  and  lessons 

In  general,  despite  the  fact  that  many  statistics  are  gross  figures  and  concerned  mostly  with  resource 
counts  and  costs,  data  collection  was  not  easy.  There  are  a number  of  issues  and  challenges  that  affect  the 
library’s  ability  to  collect  statistics  and  measures  to  describe  its  networked  resources  and  services: 

• Acquisitions , accounting,  and  cataloging  systems  are  not  set  up  to  support  this  kind  of  data 

collection.  Current  bibliographic  and  management  information  systems,  for  the  most  part,  reflect 
practices  in  the  pre-Web,  print-dominant  environment.  It  appears  that  providing  access  to 
electronic  resources  is  keeping  many  research  libraries  busy  enough  already.  The  lack  of  efficient 
information  systems  that  pull  together  elementary  data  elements  forced  many  field-testing 
libraries  to  resort  to  labor-intensive  processes  to  collect  data.  According  to  a recent  survey  done 
by  Tim  Jewell  at  the  University  of  Washington  Libraries,  there  are  about  10  ARL  libraries  that 
have  a production  system  for  managing  electronic  resources,  and  several  others  in  the  planning  or 
development  stage  (http://www.librarv.comell.edu/cts/elicensestudv/home.htmn.  While  these 
systems  are  not  developed  solely  for  data  collection  purposes,  they  certainly  facilitate  the  data 
collection  efforts  such  as  the  E-Metrics  project.  In  the  absence  of  such  fully  developed 
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information  systems,  we  advise  ARL  libraries  to  develop,  at  a minimum,  an  in-house  spreadsheet 
or  database  file  to  keep  track  of  key  data  elements  related  to  electronic  resources  and  services. 

• Prescribed  definitions  and  procedures  are  not  compatible  with  local  practices.  Several  field- 
testing  libraries  independently  have  been  collecting  some  of  the  similar  statistics  and  measures, 
but  their  definitions  and  promulgation  of  the  methodologies  differ  from  what  the  field-testing 
entailed.  It  seems  that  the  majority  of  libraries  want  to  build  their  local  procedures  in  sync  with 
the  standardized  ARL  practices,  a sentiment  that  is  echoed  in  the  following  comment:  “We  will 
adjust  our  in-house  practices  to  be  able  to  report  in  this  way.”  The  data  collection  manual 
included  in  this  report  is  one  step  in  that  direction. 

• The  nature  of  electronic  resources  and  services  is  still  fluid  and  makes  it  difficult  to  devise  clear- 
cut  definitions  and  procedures.  For  instance,  as  several  people  have  already  argued  (Snowhill, 
2001;  Sottong,  2001),  the  concept  of  electronic  books  is  still  evolving  due  to  changes  in 
technology,  the  market,  and  use  of  resources  among  other  things.  For  example,  think  of  the  full- 
text  search  capability  in  most  electronic  books.  It  can  be  argued  that  there  is  no  clear  distinction 
between  electronic  books  and  reference  sources,  especially  from  the  user’s  point  of  view.  We 
observe  that  electronic  access  can  trigger  an  entirely  new  conceptualization  of  a given  information 
object  as  in  the  case  of  electronic  books.  Libraries  need  to  deal  with  the  implications  of  this 
changing  environment  and,  as  one  librarian  commented  during  the  site  visit,  be  more  flexible.  We 
acknowledge  that  the  distinction  made  for  different  electronic  resources  in  the  manual  and  in  the 
current  E-Metrics  work  is  only  temporary  and  will  have  to  be  revised  as  we  progress. 

• The  dispersed  nature  of  resources  in  the  networked  environment  makes  it  difficult  to  consolidate 
and  manage  statistics.  It  also  is  a growing  source  of  frustration  for  many  librarians  who  deal  with 
electronic  resources.  Various  listservs  devoted  to  electronic  resources  and  voluminous 
correspondence  on  the  listservs  reflect  this  trend.  Traditionally,  library  materials,  with  the  notable 
exception  of  government  publications,  are  centrally  managed  through  a library  catalog.  Also, 
library  visit  counts  have  traditionally  been  normalized  by  using  turnstile  counts  whenever 
possible.  However,  in  the  networked  environment,  libraries  have  to  deal  with  a whole  range  of 
resources  and  access  points.  This  in  turn  creates  more  complexity  in  not  only  managing  resources 
but  also  collecting  data  about  the  resources  and  their  use.  For  example,  with  respect  to  usage 
statistics  of  licensed  materials,  while  setting  up  a library  database  gateway  may  allow  the  library 
to  collect  a coherent  statistic  (e.g.,  attempted  logins  to  licensed  databases),  it  does  not  account  for 
traffic  that  goes  directly  to  vendor  websites.  On  the  other  hand,  usage  statistics  from  database 
vendors  are  more  complete  in  the  sense  that  they  capture  all  requested  use  of  the  database,  but  the 
incompatibility  of  statistics  from  various  vendors  makes  it  difficult  for  the  libraries  to  compare 
and  aggregate  usage  data.  Therefore,  it  is  important  that  libraries  be  able  to  deal  with  incomplete, 
incompatible  data  from  multiple  sources  and  make  the  best  decisions  based  on  the  given  data. 

• Related  to  the  issue  of  the  dispersed  nature  of  resources  and  performance  statistics  is  the 
organizational  structure  needed  to  manage  electronic  resources  and  services,  particularly  the 
configuration  of personnel  and  workflow  to  support  the  collection  of  statistics  and  measures.  In 
the  context  of  field-testing,  we  observed,  to  the  extent  it  can  be  generalized,  two  types  of 
organizational  setups  used  to  carry  out  data  collection: 

1 . A single  person  making  requests  to  other  librarians  and  staff  members  for  data  needs  and 
processing  and  summarizing  collected  data.  This  person  is  most  likely  the  electronic  resource 
librarian,  head  of  collections,  or  someone  who  is  responsible  for  overall  library  statistical 
matters.  For  this  kind  of  organizational  structure  to  work,  the  library  needs  to  understand  that 
what  this  person  is  doing  is  critical,  and  to  see  the  benefits  of  sharing  information.  A 
drawback  to  this  arrangement  is  that  there  is  a lack  of  overlap  of  expertise  and  broad  sharing 
of  information. 
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2.  A team  structure  which  coordinates  the  entire  aspect  of  electronic  resource  management 
including  acquisition,  access,  and  evaluation.  An  example  of  this  kind  of  structure  is  the 
NetDOC  (Networked  Databases  on  Campus)  at  the  University  of  Manitoba  Library. 
Consisting  of  about  10  people  from  various  library  departments,  such  as  reference,  collection 
management,  systems,  and  administration,  the  committee  oversees  the  management  of 
electronic  services  through  monthly  and  yearly  reviews.  Collection  and  review  of  usage 
statistics  of  licensed  resources  is  an  integral  part  of  the  committee’s  activities.  While  the 
amount  of  staff  time  and  coordination  can  be  substantial,  it  gives  various  internal  stakeholders 
an  opportunity  to  participate  in  the  decision-making  process.  In  addition,  it  promotes  the 
sharing  of  expertise  in  the  library. 


As  previously  discussed,  collecting  data  regarding  networked  resources  and  services  is  becoming 
increasingly  dispersed  and  complex.  Libraries  need  to  consider  organizational  strategies  to  best  support 
the  management  and  monitoring  of  library  services  in  the  networked  environment. 


liaisoi 


librarians  and  staff 
members 
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3 VENDOR  STATISTICS  STATUS  REPORT 


This  part  describes  the  activities  and  accomplishments  of  the  study  team  and  the  members  of  the 
Working  Group  on  Database  Vendor  Statistics  on  standardizing  usage  statistics  from  content  providers. 

3.1  Phase  I Activities  Summary 

During  the  ARL  E-Metrics  Project's  Phase  One,  the  study  team  completed  data  collection  and  analysis 
along  a broad  range  of  activities  and  issues  related  to  measures  of  networked  information  resources  and 
services.  The  brief  overview  below  highlights  the  activities  related  to  vendor  statistics  in  Phase  I. 


During  the  July  2000  ALA  annual  conference  in  Chicago,  the  ARL  Working  Group  (WG)  on 
Database  Vendor  Statistics  had  its  first  meeting.  The  WG  is  a smaller  set  of  study  participants  (13)  that 
focuses  on  the  statistics  describing  use,  users,  costs,  etc.,  of  vendor  databases.  During  the  meeting, 
members  of  the  Working  Group  were  asked  to  submit  a list  of  vendors  with  which  they  spent  more  than 
$20,000  for  annual  subscriptions  and  also  to  supply  sample  vendor  reports  and  other  information  relevant 
to  vendor  statistics. 

The  study  team  received  the  vendor  lists  from  nine  libraries.  Among  these  libraries,  some  sent  us  the 
top  ten  vendors,  while  others  included  a fairly  comprehensive  list  of  database  vendors  with  whom  they 
have  subscriptions.  The  study  team  then  reviewed  the  lists  and  identified  the  12  most  frequently  listed. 
Figure  3.1  below  lists  the  names  of  vendors  whose  usage  statistics  reports  were  analyzed  in  this  study. 

Figure  3.1  List  of  Compared  Vendor  Reports 

Academic  Press/IDEAL  Bell  & Howell/ProQuest  Ebsco 

Elsevier/ScienceDirect  GaleNet  High  Wire 

ISI/Web  of  Science  JSTOR  Lexis-Nexis 

OCLC/FirstSearch  Ovid  SilverPlatter  


The  study  team  reviewed  the  sample  vendor  reports  sent  by  libraries  in  terms  of  the  statistics 
provided,  format  of  the  reports,  frequency  of  reporting,  statistics  access,  and  other  information  that 
provides  a concise  summary  of  vendor  offerings.  The  study  team  also  tried  to  match  statistics  from  each 
vendor  to  the  ICOLC  (International  Consortium  of  Library  Consortia)  guidelines,  as  libraries  and  vendors 
both  seem  to  be  leaning  toward  the  guideline  as  a current  yardstick  of  how  successful  a vendor  can  be  in 
delivering  usage  statistics. 

Finally,  the  study  team  reviewed  the  definitions,  descriptions,  or  explanations  of  the  usage  statistics 
provided  by  the  vendors  along  with  the  actual  reported  statistics.  This  provided  data  regarding  how  much 
agreement  there  was  among  library  vendors  as  to  what  the  reported  measures  mean.  Thus  the  findings 
presented  in  this  report  provide  a snapshot  at  a particular  point  in  time  and  serve  as  a standardized  review 
of  today’s  vendor  statistics. 
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3.1.2  COMPARISON  OF  VENDOR  REPORTS 

Comparing  vendor  statistics  reports  proved  to  be  a challenging  task.  First,  many  reports  were  written 
in  such  a way  that  it  was  difficult  to  decipher  what  exactly  the  reported  statistics  mean.  Second,  the 
reports  are  evolving  constantly  as  the  vendors  change  both  the  content  and  the  format  in  attempts  to 
improve  their  service.  The  analysis  focused  on  three  main  areas:  (1)  Overall  comparison  of  reported 
measures  and  delivery,  (2)  Availability  (or  lack)of  definitions,  and  (3)  Vendor  compliance  with  the 
ICOLC  guidelines. 

Findings  and  issues  identified  in  the  comparison  of  vendor  reports  were  as  follows: 

• Many  vendors  offer  multiple  delivery  options.  While  the  HTML  format  is  easily  viewed  in  a 
browser,  text  format  (ASCII  or  comma  separated  ) is  easier  to  manipulate  using  a spreadsheet  or 
database  program.  In  some  cases,  libraries  were  not  informed  of  the  fact  that  the  same  report  was 
available  in  an  alternative  form. 

• While  most  other  reports  provide  raw  statistics  only \ JSTOR  provides  institutional  comparison  of 
usage  based  on  the  size  of  institutions.  It  also  provides  graphs  showing  usage  trends  over  time. 

• Although  not  included  in  the  comparison , the  timeliness  of  usage  reports  was  identified  as  one  of 
the  key  issues.  While  the  majority  of  the  vendors  who  provide  a monthly  report  do  so  within  the 
following  month,  some  vendors  lag  several  months  behind  and  in  some  cases  reports  for  a block 
of  months  are  not  available  due  to  the  technical  failures  of  the  vendor  reporting  program.  It  is 
strongly  recommended  that  the  availability  of  timely  (needs  to  be  specified)  reports  be  stipulated 
in  the  licensing  contract. 

• When  database  vendors  provide  journal  level  statistics,  they  should  be  arranged  in  such  a way 
(e.g.,  alphabetically)  to  facilitate  easy  manipulation.  If  a database  product  or  a journal  is  broken 
into  several  different  parts,  as  in  the  case  of  Medline  having  several  blocks  of  files  divided  by  the 
database  coverage,  the  total  usage  for  that  database  or  journal  needs  to  be  provided. 

• When  the  usage  statistics  are  available  on  the  web,  a user  ID  and  a password  are  required  to 
access  the  statistics.  JSTOR  is  an  exception.  It  does  not  require  a user  ID  and  password.  Instead 
anyone  whose  IP  address  belongs  to  the  eligible  EP  block  for  use  authentication  can  request  and 
retrieve  statistics.  Sometimes,  additional  ID’s  and  passwords  are  required  to  access  statistics  for 
each  database  or  journal.  HighWire  is  an  example  of  such  a case.  Currently,  it  requires  libraries 
to  use  the  same  user  ID  and  password  to  access  statistics  as  well  as  administer  institutional 
profiles. 

• Overall,  libraries  want  usage  statistics  that  are  easy  to  obtain  and  manipulate  if  necessary.  In 
other  words,  there  should  be  little  burden  on  the  library  to  process  the  statistics.  Ideally,  vendors 
need  to  use  standardized  measures  and  formats.  Another  issue  has  to  do  with  different  needs  from 
different  libraries.  It  is  strongly  recommended  that  a summary  level  information  be  provided  to 
all  libraries.  At  the  same  time,  detailed  and  granular  statistics,  in  ASCII  (Text)  or  comma 
separated  file  (CSV)  format,  should  also  be  available  to  libraries  that  wish  to  obtain  additional 
information  about  use  and  use  patterns. 

• One  of  the  most  frequently  cited  complaints  about  the  current  vendor  reports  is  the  lack  of 
standardized  terms  for  key  usage  indicators  and  the  lack  of  definitions.  Even  when 
documentation  is  provided,  it  often  consists  of  simple  paraphrases  of  measures  rather  than 
explanatory  descriptions  of  what  the  measures  stand  for  and  how  they  are  being  counted  by  the 
vendor. 
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In  conclusion,  the  analysis  of  usage  statistics  from  12  major  database  vendors  revealed  a wide  range 
of  different  practices.  Details  of  these  differences  are  discussed  in  the  Phase  I report.  Progress  is  needed 
in  several  areas  including  standardization  of  core  statistics,  method  of  report  delivery,  and  the  timely 
provision  of  definitions  of  reported  statistics.  Some  signs  indicate  that  some  vendors  are  moving  in  the 
right  direction.  One  of  the  signs  is  the  apparent  willingness  of  many  vendors  to  comply  with  the  ICOLC 
guidelines.  Coordination  of  these  efforts  seems  necessary  to  bring  about  positive  change  in  reporting 
practices  and  to  develop  consistent  and  comparable  statistics. 


3.2  ARL  Meeting  with  Vendors 

The  E-Metrics  Phase  I study  of  vendor  statistics  provided  an  overall  picture  of  the  current  state  of 
usage  statistics  offerings.  However,  the  study  team  felt  that,  to  make  progress  in  the  area,  we  needed  to 
work  with  database  vendors  directly.  Due  to  the  large  number  of  database  vendors  to  which  typical  ARL 
libraries  subscribe,  the  study  team  decided  to  focus  on  a select  group  of  large  database  vendors  that  were 
identified  during  Phase  I. 

A meeting  was  planned  on  March  14,  2001,  in  conjunction  with  the  ACRL  Annual  Meeting  in 
Denver.  Sherrie  Schmidt  (Arizona  State)  and  Rush  Miller  (University  of  Pittsburgh),  Project  Co-Chairs 
sent  invitations  on  behalf  of  ARL  to  the  twelve  vendors  analyzed  in  Phase  I plus  netLibrary. 

The  meeting's  goal  was  to  engage  the  community  of  vendors,  publishers  and  libraries  in  building 
consensus  for  reporting  data  on  the  use  of  vendor  database  statistics  for  libraries  and  promote 
understanding  of  what  can  and  cannot  be  done  vis  a vis  the  provision  of  data  from  the  vendor  community. 
The  meeting  served  as  a discussion  forum  for: 

• Sharing  information  about  the  development  and  standardization  of  selected  statistics  that  describe 
users  and  uses  of  databases; 

• Reaching  agreement  on  the  important  data  elements  and  definitions; 

• Engaging  vendors  in  a test  of  data  elements  the  study  team  designed; 

• Understanding  the  issues  that  affect  vendor  supplied  statistics  describing  data  base  use  and  users; 
and 

• Developing  a process  so  that  the  library  community  and  the  vendor  community  can  work  together 
in  developing  and  standardizing  a core  set  of  statistics. 


A total  of  nine  vendors  (Figure  3.2)  attended  the  meeting.  All  of  the  vendors,  except  for  netLibrary, 
were  the  ones  the  study  team  identified  during  Phase  I analysis. 


Figure  3.2  List  of  Database  Vendors  attended  the  ARL  Meeting 


Elsevier/ScienceDirect 

netLibrary 

OCLC/FirstSearch 

JSTOR 

Bell  & Howell 

Ovid 

Lexis-Nexis 

Gale  Group 

EBSCO 

From  the  E-Metrics  Project,  the  following  people  attended  the  meeting: 

• Co-chairs:  Sherrie  Schmidt  (Arizona  State)  and  Rush  Miller  (Pittsburgh); 

• Library  Representatives:  Betty  Day  (Maryland),  Kurt  Murphy  (Arizona  State),  and  Sue  Phillips 
(Texas-Austin,  also  ICOLC  liaison); 
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• Study  team:  Wonsik  “Jeff 5 Shim  (Florida  State);  and 

• ARL  Staff:  Martha  Kyrillidou  and  Julia  Blixrud. 

The  following  summarizes  the  events  during  the  meeting. 


• Rush  Miller  gave  a brief  overview  of  the  ARL  E-Metrics  project. 

• Sue  Phillips  reported  that  an  initial  survey  is  under  way  to  find  out  the  ICOLC  member  libraries' 
opinions  about  vendor  statistics.  The  results  will  be  used  as  a basis  to  update  the  ICOLC 
guidelines. 

• Jeff  Shim  provided  some  background  information  on  why  libraries  need  usage  data  from 
electronic  database  vendors,  and  what  kinds  of  data  they  need;  he  also  gave  an  illustration  of  data 
use  in  one  project  participant  library.  The  data  presented  were  all  based  on  the  E-Metrics  study 
results  so  far. 

• Vendors  presented  their  current  and  future  report  capabilities  and  offerings  followed  by  a 
question-and-answer  session. 

• Jeff  Shim  talked  about  the  current  state  of  vendor  statistics  using  the  findings  from  the  E-Metrics 
project.  He  also  identified  issues  that  need  to  be  worked  out  in  the  future. 

• Jeff  outlined  the  next  steps  in  terms  of  short-term  and  long-term  goals,  which  included  sending  a 
list  of  field-testing  libraries  and  more  specific  information  (tasks,  requirements)  to  the  vendors. 
The  vendors  interested  in  participating  in  the  field-testing  were  asked  to  work  with  the  study  team 
and  the  field-testing  libraries  during  the  month  of  May  2001. 

Some  of  the  issues  discussed  during  the  meeting  included  the  following: 

• Both  vendor  and  library  representatives  agreed  that  the  reported  statistics  should  be  based  on  the 
ICOLC  guidelines. 

• There  was  a concern  that  the  ICOLC  guidelines  were  composed  with  electronic  databases  in 
mind.  Many  vendors  now  have  mixed  content,  and  it  appears  that  there  aren’t  many  guidelines 
for  these  vendors. 

• Related  to  the  previous  item,  the  market  is  increasingly  diversified  in  terms  of  business  models, 
contents  provided  by  vendors,  and  so  on.  Accordingly,  developing  a standardized  set  of  statistics 
that  cover  all  of  these  is  going  to  be  a big  challenge. 

• Everyone  agreed  that  technologies  and  technology  changes  have  a lot  to  do  with  what  and  how 
statistics  can  be  collected  and  reported.  For  instance,  Z39.50  clients  do  not  allow  statistics  to  be 
collected.  There  are  solutions  that  are  also  technology  based,  such  as  digital  certificates. 
However,  in  most  cases,  the  costs  of  buying  and  implementing  these  technologies  do  not  justify 
their  use  to  produce  more  reliable  and  detailed  data. 

• It  appears  that  different  counting  mechanisms  are  in  place  at  different  vendors.  So  the  same 
statistic  is  counted  differently.  There  should  be  some  investigation  into  whether  the  reported 
statistics  are  comparable. 

Overall,  the  meeting  was  very  useful  in  the  sense  that  it  brought  two  parties,  libraries  and  vendors, 
together.  In  addition,  it  was  a necessary  step  for  the  upcoming  field-testing.  As  a result  of  the  meeting, 
all  of  the  vendors  present  agreed  to  participate  in  the  vendor  statistics  field-testing. 


ARL  E-Metrics  Phase  II  Report 


46 


October  2001 


Measures  and  Statistics  for  Research  Library  Networked  Services 


3.3  Vendor  Statistics  Field-testing 

This  section  describes  the  vendor  statistics  portion  of  the  E-Metrics  field-testing.  The  primary  goal  of 
the  field-testing  was  to  assess  usage  statistics  from  major  database  vendors  in  terms  of  comparability  of 
statistics  and  their  definitions,  breakdown  of  data  and  report  formats. 


3.3.1  Vendor  Statistics  Field-Testing  Methodologies 


All  twelve  vendors  agreed  to  participate  in  the  field-testing.  The  invitation  explained  the  goals  and 
objectives  of  the  field-testing  and  provided  a brief  summary  of  expected  deliverables  from  each 
participating  vendor. 

The  study  team  developed  a set  of  field-testing  guidelines  and  distributed  an  electronic  copy  to  the 
vendors.  In  addition,  the  study  team  contacted  project  participants  and  solicited  their  participation  in  the 
field-testing.  Since  not  all  field-testing  libraries  subscribed  to  all  of  the  services,  the  study  team  assigned 
three  or  four  vendors  to  each  library,  as  shown  in  Table  3.1,  based  on  the  libraries’  subscription  matrix. 
The  intent  was  to  alleviate  the  burden  on  the  libraries  of  evaluating  too  many  vendor  reports.  Also,  from 
the  standpoint  of  vendors,  it  seemed  to  make  sense  to  concentrate  on  a few  libraries  rather  than  all  of  the 
libraries  subscribing  to  their  services. 


Table  3.1  Library  Assignment  for  Vendor  Statistics  Field-Testing 


Academic 

Press 

Bell  & 
Howell 

Ebsco 

Science 

Direct 

Gale 

Net 

ISI/  Web 
Science 

JSTOR 

Lexis- 

Nexis 

Net 

Library 

OCLC/ 
Firsts  earch 

OVID 

Silver 

Platter 

Alberta 

XXX 

xxx 

xxx 

Arizona  State 

XXX 

XXX 

XXX 

Auburn 

XXX 

XXX 

xxx 

Cornell 

XXX 

xxx 

XXX 

XXX 

Manitoba 

xxx 

xxx 

XXX 

Maryland 

xxx 

xxx 

XXX 

Massachusetts 

XXX 

xxx 

xxx 

Pennsylvania 

xxx 

xxx 

xxx 

Pittsburgh 

xxx 

xxx 

XXX 

xxx 

Texas-Austin 

xxx 

XXX 

xxx 

xxx 

Western 

Ontario 

xxx 

xxx 

XXX 

Virginia  Tech 

XXX 

XXX 

xxx 

Yale 

xxx 

xxx 

xxx 

Library  assignment  is  shown  as  triple  x marks  (xxx). 


The  guidelines  specifically  asked  the  following  deliverables  from  each  vendor: 

• A monthly  report  (April  2001)  in  a standardized  text  format  (specific  guidelines  were  given  for 
data  elements  and  their  arrangement); 

• A detailed,  step-by-step  description  of  the  process  employed  to  collect  the  statistics  including  the 
rules  and  assumptions  applied  in  the  process; 

• A monthly  (April  2001)  raw  data  log  file;  and 

• Issues  and  suggestions  related  to  providing  usage  statistics. 
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The  study  team  asked  the  vendors  to  send  the  field-testing  data  to  their  assigned  libraries  as  well  as  to 
the  study  team  by  the  last  week  of  May. 

The  study  team  asked  the  field-testing  libraries  to  assess  each  vendor  report  considering  the  following 
questions. 

1.  Describe  what  data  and  information  you  received  from  the  vendor  during  the  field-testing. 
Describe  the  file  format,  contents,  data  elements,  description,  and  organization  of  the  data  as 
received. 

2.  What  level  of  effort  was  required  to  manipulate  the  field-testing  monthly  report  data  you  received 
from  the  vendor?  How  did  you  process/analyze  the  data  (e.g.,  tools,  analysis,  etc)? 

3.  Please  describe  how  the  field-testing  monthly  report  differed  from  the  reports  you  usually  receive 
from  the  vendor. 

4.  Were  you  able  to  compare  the  data  fields  accurately  to  other  data  fields  provided  by  this  vendor  as 
well  as  to  data  fields  from  other  vendor  reports?  Describe  specific  issues/problems  you 
encountered  in  doing  so. 

5.  To  what  degree  are  the  data  from  this  report  worth  using  for  library  decision-making  given  the 
amount  of  time  and  effort  required  to  analyze  them? 

6.  What  recommendations  would  you  make  to  improve  the  vendor  report? 

3.3.2  Vendor  Statistics  Field-testing  Results 
3.3.2.1  Vendor  Report  Comparison 


A total  of  eight  vendors  were  actually  able  to  participate  in  the  field-testing,  as  shown  in  Table  3.2. 
The  table  also  shows  the  data  formats  in  which  the  field  test  report  were  provided  by  the  vendors  and  the 
availability  of  documentation  received  from  the  vendors  in  regard  to  definitions  of  statistics  provided  and 
information  about  how  data  are  collected,  filtered  and  aggregated. 


Table  3.2  Vendor  Statistics  Field-testing  Participation 


Vendors 

Data  Format 

Availability  and  quality  of  Documentation 

Definition 

Data  collection 

Academic  Press 

txt,  Excel 

n/a 

n/a 

Bell  & Howell 

zip(Excel,  txt,  PDF)* 

Fair 

Poor 

Ebsco 

txt 

Good 

Fair 

Gale  Group 

CSV 

Fair 

Poor 

Lexis-Nexis 

zip(csv),Word,  txt 

Poor 

Fair 

NetLibrary 

zip(txt),  csv 

Good 

Good 

Science  Direct 

txt 

Fair 

Good 

SilverPlatter 

CSV 

n/a 

n/a 

ISI  (Web  of  Science),  JSTOR,  OCLC  (First  Search)  and  Ovid  did  not  participate  in  the  field-testing. 

* Part  of  report  in  PDF  format;  n/a:  not  available  from  the  vendor  during  the  field-testing. 

Poor:  shallow,  not  specific;  Fair:  somewhat  useful;  Good:  easy  to  understand  , useful;  Excellent:  in-depth,  easy  to 
understand. 


While  it  was  one  of  the  requirements  in  the  vendor  statistics  field-testing,  the  majority  of  vendors  we 
investigated  during  Phase  II  provide  usage  reports  in  a text  format  as  well  as  other  formats.  However,  as 
we  will  discuss  later,  standardization  of  data  elements  and  their  arrangement  are  key  areas  for  further 
improvement. 
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As  compared  to  the  results  from  the  Phase  I vendor  statistics  analysis,  the  evidence  indicates  that 
vendors  have  made  good  efforts  especially  in  the  area  of  making  documentation  available.  Many  vendors 
simply  did  not  have  any  documentation  about  usage  statistics  at  all  when  we  analyzed  their  reports  during 
Phase  I.  Several  vendors’  documentation  was  organized  like  their  report  structures  and  did  not  contain  the 
information  that  met  the  requirements  for  the  field-testing. 


Table  3.3  Key  ICOLC  Statistics  Included  in  the  Vendor  Reports  (by  vendor) 


VENDORS 

Items  requested 

Searches 

Sessions 

Tumaways 

Academic  Press/ 
IDEAL 

Fulltext,  reference,  abstract, 
table  of  contents 

Yes 

Yes 

n/a 

Bell  & Howell 

Fulltext,  abstract,  citation 

Yes 

No 

n/a 

Ebsco 

Fultext,  abstract 

Yes 

Yes 

n/a 

Gale  Group 

Fulltext,  citation  & abstract, 
hits,  views,  print  station 

Yes 

Yes 

Yes 

Lexis-Nexis 

Fulltext,  document  retrievals 

Yes 

No 

n/a 

NetLibrary 

Pageview,  browse,  checkout, 
dictionary  use 

Yes 

Yes 

Yes 

Science  Direct 

Fulltext,  abstract 

Yes 

Yes 

n/a 

SilverPlatter 

Fulltext,  abstract 

Yes 

Yes 

Yes 

n/a:  not  applicable 


Table  3.3  shows  key  ICOLC  statistics  included  in  each  vendor’s  field-testing  report.  The  study  team 
has  not  attempted  to  validate  the  compliance  with  the  ICOLC  guidelines.  Aside  from  the  ICOLC 
guidelines,  there  are  many  instances  where  the  same  statistics  from  different  vendors  are  not  really  equal 
measures.  For  example,  the  session  count  will  vary  considerably  depending  on  the  time-out  cutoff  that 
vendors  use.  The  documentation  shows  a wide  range  of  time-outs  (e.g.,  Gale:  6 minutes,  Ebsco:  10 
minutes,  and  Science  Direct:  30  minutes). 

Another  area  that  makes  cross  comparison  more  difficult  is  the  items  requested  statistic,  due  to  the 
fact  that  the  types  of  content  available  through  vendors  are  diverse  and  the  terms  referring  to  information 
items  are  not  quite  standardized.  For  example,  netLibrary,  who  was  not  studied  during  Phase  I but 
nonetheless  is  gaining  noticeable  presence  in  research  libraries,  does  not  lend  itself  easily  to  the  kinds  of 
statistics  we  are  now  familiar  with.  This  poses  a serious  problem  when  libraries  try  to  aggregate  the  total 
number  of  item  access  for  cross  vendor  comparison  or  to  gauge  the  total  amount  of  information  transfer 
from  licensed  materials  available  at  their  institutions. 

The  tum-away  statistic  has  been  useful  in  determining  whether  or  not  to  increment  (rarely,  to  reduce) 
the  number  of  simultaneous  user  licenses.  Table  3.4  shows  that  out  of  the  eight  vendors,  only  three  have 
such  a restriction. 
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Table  3.4  Breakdown  of  Statistics  in  the  Vendor  Reports  (by  vendor) 


Vendors 

By  journal  or  database 
title 

IP 

Time/Day 

Other 

Academic  Press/ 
IDEAL 

Journal  title 

Yes 

n/a 

Bell  & Howell 
/Proquest 

Database  title,  journal  title 

Yes 

Time 

Client  ID 

Ebsco 

Database  title 

Yes 

n/a 

Group  and  profile  ID 

Gale  Group 

Database  title,  journal  title 

n/a 

Time,  Day 

Lexis-Nexis 

Database  title 

Yes 

Time,  Day* 

NetLibrary 

Book  title 

Yes 

Time,  day 

Science  Direct 

Journal  title 

Yes 

n/a 

Subscribed  vs.  non- 
subscribed 

SilverPlatter 

Database  title 

n/a 

n/a 

Peak  time  and  duration 

By  journal  title  (or  journal  name  by  some  vendors)  means  statistics  are  reported  at  the  journal  or  newspaper  or 
magazine  level;  by  database  title  means  statistics  are  reported  at  groups  of  journals  or  source  files  designated  by  the 
vendors. 

n/a:  not  available,  * average  usage 

Table  3.4  shows  a breakdown  of  reported  statistics  according  to  the  ICOLC  recommended  categories. 
It  also  lists  any  other  breakdown  categories  that  the  vendors  reported.  It  appears  that  vendors,  in  general, 
satisfied  the  title  level  (journal,  database,  or  book)  breakdown  requirement.  The  IP  breakdown 
requirement  was  also  being  respected.  But  in  all  cases,  the  statistics  were  lumped  at  the  subnet  (a  group 
of  IP  address  block)  level  at  the  individual  IP  address  level.  The  tabulation  might  not  have  been  included 
in  summary  statistics  anyway  because  it  is  something  that  can  be  available  in  log  files.  Unfortunately, 
most  vendors  were  unable  to  furnish  log  data  files  due  to  technical  and  legal  concerns.  Half  of  the 
vendors  currently  provide  some  time-related  breakdowns. 

3.3. 2. 2 Libraries'  Evaluation  of  Vendor  Reports 

3.3. 2. 2.1  Level  of  effort  required  to  manipulate  the  field-testing  monthly  reports 

Overall,  libraries  reported  that  the  data  files  were  easy  to  read  and  process.  The  majority  of  libraries 
used  Microsoft  Excel  to  import  and  display  data  files.  In  one  case,  a vendor  sent  part  of  the  data  files  in 
pdf  format  and  forced  the  recipient  libraries  to  enter  the  numbers  manually.  The  results  show  that 
libraries  would  prefer  data  formats,  notably  text  formats,  that  can  be  easily  imported  into  data  analysis 
programs  such  as  Excel  and  Lotus  1-2-3  without  having  to  spend  extra  time  and  effort  to  manipulate  or 
enter  data. 

While  all  participating  libraries  at  least  opened  the  data  files,  only  a few  attempted  to  analyze  the  data. 
There  seemed  to  be  several  reasons  why  libraries  were  hesitant  about  in-depth  analysis  of  data.  One 
library  commented  that  it  did  not  test  the  data  since  it  was  the  summary  data  and  not  raw  data  which  they 
expected  from  the  field-testing.  The  following  comment  from  another  library  also  explains  why  libraries 
have  not  done  further  analysis:  “We  currently  place  raw  vendor  statistics  on  our  staff  intranet  and  do  not 
compile  them  for  comparison  purposes,  as  we  have  yet  to  define  what  statistics  and  what  format  would 
best  suit  our  institutional  needs  for  such  a compilation.” 

At  least  one  library  reported  specifically  how  they  processed  the  field-testing  data.  For  each  vendor 
they  analyzed,  they  compared  the  session  counts  from  the  library  redirect  page  (all  requests  to  external 
vendor  databases  pass  through  a webpage  that  counts  how  many  times  different  databases  are  accessed) 


ARL  E-Metrics  Phase  II  Report 


50 


October  2001 


O 

ERIC 


195 


Measures  and  Statistics  for  Research  Library  Networked  Services 


and  the  vendor  report.  This  produced,  for  each  database,  a rough  idea  of  what  portion  of  attempted  logins 
(sessions)  originates  from  people  who  bypass  the  library  database  webpage.  The  library  also  calculated 
the  estimated  cost  per  article  viewed  and  distribution  of  articles  viewed  by  title,  which  confirmed  that 
25%  of  the  titles  account  for  80%  of  articles  viewed  for  the  particular  database. 

3.3. 2.2. 2 Comparison  of  field-testing  report  and  regular  monthly  report  that  libraries 
received 

The  field-testing  instructions  provided  guidelines  in  terms  of  essential  data  elements,  data 
arrangement  and  file  format.  Contrary  to  our  expectations,  all  of  the  vendors  simply  repackaged  their 
monthly  usage  reports  and  submitted  them  to  the  libraries  and  the  study  team  as  their  field-testing  report. 
Therefore,  the  only  practical  difference  between  the  field-testing  report  and  the  report  that  libraries  access 
from  the  vendor  website  in  a normal  situation  was  that  libraries  received  the  data  files  directly  from  the 
vendors  instead  of  retrieving  them  from  vendor  websites.  Several  libraries  appreciated  the  fact  that  they 
could  receive  data  files  in  text  format,  which  is  much  easier  to  handle  than,  say,  HTML  format.  Another 
minor  difference  was  the  availability  of  data  definitions  and  statistics  collection  processes  from  some  of 
the  participating  vendors.  In  some  cases,  this  was  the  first  time  that  the  explanations  were  available  to  the 
libraries.  Typically  documents  that  contain  definitions  of  statistics  and  other  background  information,  if 
they  are  available,  are  provided  on  the  vendors’  websites. 

3.3. 2.2.3  Comparison  of  field-testing  data  with  the  same  vendors  and  across  vendors 

We  assigned  three  or  four  vendors  to  each  field-testing  library  so  that  libraries  could  compare  data 
from  multiple  vendors.  Largely  it  did  not  work  as  we  had  intended  because  several  vendors,  who 
promised  to  participate,  did  not  provide  data  at  all. 

Even  if  data  were  available  from  several  vendors,  it  was  difficult  for  the  libraries  to  do  valid 
comparisons  of  data  because  sufficient  descriptions  of  data  definitions  and  how  the  data  were  collected 
and  summarized  were  not  available.  Many  libraries  feared  that,  without  the  explanatory  information 
about  what  each  data  element  in  vendor  reports  meant  and  how  the  counts  were  filtered,  such  comparison 
would  have  resulted  in  comparing  apples  and  oranges.  This  suggests  that  until  there  is  a satisfactory 
degree  of  assurance  that  the  statistics  different  vendors  provide,  based  on  the  documentation  they  provide, 
are  consistent  enough  for  cross  comparison,  libraries  will  not  commit  major  resources  to  compile  vendor 
data  into  a standardized  format  or  repository. 

Another  problem  with  comparing  data  from  multiple  vendors  was  the  inconsistent  data  formats.  The 
task  of  combining  data  fields  and  adjusting  data  arrangement  from  even  three  or  four  vendors  proved  to  be 
extremely  time  consuming.  What  libraries  want  is  a standardized  usage  report,  containing  common  data 
elements  and  arranged  in  predetermined,  agreed  upon  order,  that  is  provided  separately  from  vendor 
specific  data  elements  or  additional  data.  Even  the  different  placement  of  field  headings,  in  a column  or 
in  a row,  requires  extra  efforts  from  the  libraries. 

3.3. 2. 2.4  Value  of  Vendor  Reported  Usage  Data 

The  majority  of  respondents  said  that  the  data  provided  by  these  vendors  are  “necessary  and 
valuable”.  They  like  the  fact  that  the  data  are  “very  straightforward  and  easy  to  use”  and,  more 
importantly,  that  the  data  provide  some  indication  of  the  extent  to  which  subscription-based  services  are 
being  utilized.  Of  course,  the  relative  value  depends  on  the  quality  of  data  and  the  importance  of  the 
database  to  the  library  (e.g.,  the  amount  of  money  the  library  spends  for  a particular  database  as  compared 
to  other  databases  they  subscribe  to). 
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While,  the  majority  of  libraries  feel  that  usage  reports  provided  by  the  individual  vendors  are  useful, 
there  is  some  doubt  about  the  cumulative  value  of  all  usage  reports  combined.  Given  the  fact  that  typical 
ARL  libraries  deal  with  several  dozen  database  vendors,  normalizing  data,  in  current  forms,  from  these 
vendors  will  require  a considerable  effort  on  the  part  of  libraries. 

Usage  reports  almost  exclusively  deal  with  specific  use  of  vendor  databases  in  terms  of  frequencies 
(e.g.,  searches  and  sessions),  duration  (e.g.,  connection  time),  and  amount  of  information  transfer  (e.g., 
items  requested)  while  largely  ignoring  another  dimension  that  many  libraries  consider  very  important: 
information  on  user  behavior.  The  current  usage  metrics  do  provide  information  about  user  behavior  to  a 
degree  but  not  at  the  level  many  libraries  would  hope.  At  least  one  statistic  included  in  the  ICOLC 
guideline  seems  to  deal  with  an  aspect  of  user  behavior:  menu  selection.  However,  because  of  different 
interfaces  and  types  of  content  provided  by  the  vendors,  it  would  be  difficult  to  imagine  vendor  report 
menu  selection  reported  in  a consistent  manner.  As  a result,  libraries  get  bits  and  pieces  of  information  in 
as  many  different  ways  as  there  are  vendors. 

To  be  truly  useful,  information  on  user  behavior  will  need  to  be  correlated  with  individual  user 
profiles.  But  the  current  environment  for  database  access,  which  is  heavily  rooted  on  IP-based 
authentication,  does  not  permit  the  kinds  of  data  collection  that  libraries  expect.  So,  on  the  one  hand, 
there  is  a desire  to  receive  more  detailed  information  about  user  behaviors,  but,  on  the  other  hand,  that 
desire  conflicts  with  the  current  practices  and  the  libraries’  concern  about  user  privacy. 

The  best  the  vendors  can  do  is  provide  an  option  that  allows  libraries  to  access  raw  data  log  files  that 
have  sufficient  information  for  useful  analysis.  Unfortunately,  many  vendors  were  not  able  to  provide  log 
data  files  because  of  technical  and  legal  concerns. 

3.3. 2.3  Summary  of  Vendor  Statistics  Field-Testing 

3. 3. 2.3.1  Are  the  data  provided  to  libraries  reliable? 

Since  the  field-testing  dealt  with  only  one  month’s  worth  of  data,  it  is  difficult  to  answer  the  question. 
However,  we  have  not  heard  from  the  field-testing  libraries  of  any  unusual  discrepancy  between  the  field- 
testing  data  and  data  they  received  before  the  field-testing.  The  study  team  realizes  that  just  comparing 
data  from  the  same  vendors  will  not  give  us  a satisfactory  answer.  During  the  course  of  writing  this 
report,  we  came  across  an  email  message  from  a major  database  vendor  acknowledging  errors  in  their 
usage  reports.  This  suggests  that  libraries  are  really  not  in  a good  position  to  know  what  exactly  goes  into 
the  vendor  reports.  Some  unusual  numbers  or  patterns  are  relatively  easy  to  identify.  But  consistent 
under  (or  over)  counts  are  harder  to  detect. 

One  way  to  deal  with  the  reliability  issue  is  to  collect  data  that  libraries  can  generate  in-house.  For 
example,  some  libraries  have  set  up  a redirect  webpage  for  external  databases  to  count  the  number  of 
attempted  logins  to  licensed  databases.  This  kind  of  data  gives  clues  that  can  be  used  to  cross-check 
vendor-supplied  numbers.  Also,  the  library  community  needs  to  consider  concrete  ways  (e.g.,  third  party 
validation)  to  ensure  consistent  and  reliable  reporting  from  vendors,  or  at  least  should  demand  from 
vendors  better  documentation  of  the  data  collection  and  filtering  process. 

3. 3. 2.3.2  Are  the  data  comparable  across  libraries,  products,  and  vendors? 

Use  of  different  system  parameters  (e.g.,  time-out),  application  of  different  assumptions  about  user 
behavior  (e.g.,  how  to  treat  or  count  multiple  clicks  on  the  same  document  within  a session),  and  the  lack 
of  adequate  explanation  in  vendor  documentation  regarding  specific  definitions  and  data  collection  and 
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filtering  processes  all  contribute  to  the  problem.  Therefore,  we  conclude  that  it  is  largely  impossible  to 
compare  data  across  vendors,  and  we  recommend  that  comparison  be  limited  to  data  from  the  same 
vendors.  We  believe  that  the  comprehensive  standardization  of  usage  statistics  and  data  delivery  methods 
(e.g.,  file  format  and  data  arrangement)  cannot  be  easily  achieved  in  the  short-term.  Those  are  long-term 
goals  toward  which  vendors  and  libraries  need  to  work  together.  The  ARL  community  should  continue  to 
make  progress  in  this  area  by  working  amongst  themselves  and  with  the  database  vendor  community. 

3.3.2. 3.3  Is  the  data  easy  to  obtain  and  manipulate? 

We  believe  that  the  data  provided  from  the  vendors  studied  are  easy  to  obtain  and  manipulate.  Most 
vendors  offer  several  data  formats  including  text  format  (e.g.,  comma  separated  file)  and  spreadsheet 
format  (e.g.,  MS  Excel)  in  addition  to  standard  HTML  format  for  easy  viewing  in  web  browsers.  Also, 
many  vendors  offer  an  ad-hoc  report  generation  facility  whereby  libraries  can  customize  the  fields  they 
want  to  examine  and  set  desired  time  periods. 

However,  we  estimate  that  processing  vendor  reports  from  multiple  vendors  can  become  a 
considerable  burden  on  libraries,  in  terms  of  time  and  staff  efforts,  as  the  formats  and  data  arrangements 
vary  considerably  from  vendor  to  vendor.  We  strongly  recommend  that  vendors  report  standardized 
usage  statistics,  such  as  the  ones  recommended  by  the  ICOLC  and  those  found  in  part  4.  These  should 
appear  in  the  standardized  column  and  row  arrangements  and  include  a separate  report  that  contains 
vendor  specific  additional  data. 

3. 3.2. 3.4  Does  the  data  provide  meaningful  information  about  the  usage  of 
networked  information  resources? 

Usage  statistics  currently  being  provided  by  vendors  give  useful  information  regarding  the  utilization 
of  external  subscription-based  information  services.  Libraries  use  data  for  a variety  of  purposes:  usage 
trends  over  time,  justification  for  expenditures,  cost  analysis,  modification  of  service  provision.  Related 
to  the  issue  of  the  value  of  data  is  trustworthiness  (reliability)  of  data.  Also,  there  is  some  concern  over 
the  lack  of  user-related  information  in  usage  statistics  as  discussed  earlier. 

3.3.3  Short-term  Recommendations 

This  section  offers  several  suggestions  that  may  be  useful  for  ARL  libraries  to  consider  in  dealing 
specifically  with  vendor  statistics. 

• Focus  data  analysis  on  high  impact  databases:  We  recommend  that  libraries  not  treat  all 
databases  equally  when  it  comes  to  data  analysis.  Due  to  inconsistencies  of  data  elements  and 
report  delivery,  it  is  difficult  to  normalize  usage  statistics  from  all  vendors  who  report  data. 
Instead,  libraries  need  to  investigate  the  usage  patterns  of  “major”  databases,  whatever  those 
might  be  locally,  and  ways  in  which  improvements  can  be  made  in  terms  of  access  and  use  of 
materials. 

• Collect  locally  obtainable  data  for  external  databases:  While  libraries  need  to  depend  on 
database  vendors  for  usage  statistics,  they  have  several  ways  (e.g.,  through  redirect  page  counters 
for  licensed  databases  or  through  proxy  server  logs)  to  capture  at  least  partial  information  on  user 
access  to  the  external  databases  (e.g.,  attempted  logins).  This  kind  of  internal  data  helps  libraries 
spot  check  the  reliability  of  vendor-supplied  usage  statistics.  Furthermore,  since  the  data  will  be 
under  the  control  of  libraries,  they  are  more  consistent  than  measures  reported  by  different 
vendors. 
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• Keep  track  of  aggregate  key  statistics  and  use  them:  Libraries  often  find  themselves  in  need  of 
gross  figures  of  user  access  to  external  licensed  databases  for  various  internal  and  external  use. 
The  aggregate  numbers  are  good  indicators  of  overall  trends  in  user  demand  for  and  access  to 
external  databases.  In  doing  so,  it  is  important  to  keep  some  level  of  consistency  in  the  way  the 
gross  figures  are  calculated  and  reported.  One  way  to  maintain  consistency  is  to  gather  data  from 
the  same  pool  of  database  vendors  or  database  titles  over  a specified  period  of  time  (e.g.,  “Total 
number  of  searches  conducted  in  existing  licensed  databases  grew  by  20%  in  2000  to  1,200,000 
as  compared  to  the  1999  total  of  1,000,000  searches.  The  data  is  based  on  the  same  35  vendors 
who  report  the  statistic.”). 

3.4  Issues  and  Future  Actions 

ARL  libraries  have  needed  consistent,  comparable,  easy-to-use,  and  useful  usage  statistics  from 
content  providers  (database  vendors)  ever  since  they  embraced  the  notion  of  courting  their  users  of 
external  licensed  materials.  The  ARL  E-Metrics  project  provided  an  opportunity  for  the  ARL  community 
to  look  at  the  problems  related  to  vendor  usage  reporting  in  a more  systematic  way  and  to  begin  working 
toward  developing  more  useful  reports. 

Some  of  the  issues  that  need  to  be  considered  in  dealing  with  vendor  usage  reports  in  the  future 
include  the  following: 

• The  market  for  electronic  content  providers  is  becoming  more  diverse  and  complicated.  We  need 
to  think  about  the  types  of  statistics  that  best  serve  libraries  in  this  changing  environment. 
Companies  such  as  netLibrary  did  not  even  exist  when  the  ICOLC  guidelines  were  drafted.  A 
related  issue  is  the  effect  of  mega-mergers  that  are  taking  place  in  the  electronic  content 
providers’  market. 

• For  the  most  part,  we  have  been  relying  on  the  ICOLC  guidelines  as  the  de  facto  standard  for 
usage  statistics  for  licensed  materials.  While  most  vendors  included  in  the  study  claimed  a high 
level  of  compliance  with  the  guidelines,  some  librarians  remain  skeptical,  citing  the  differences  in 
the  way  statistics  are  being  collected  by  different  vendors  (e.g.,  different  time-outs)  and  the  lack 
of  concrete  documentation.  The  ICOLC  guidelines  are  concerned  mainly  with  defining  basic 
usage  statistics  and  do  not  contain  detailed  information  that  we  can  use  to  validate  vendor  reports 
adhering  to  the  standard.  Among  librarians,  we  may  have  different  opinions  about  how  statistics 
should  be  counted.  What  level  of  specificity  are  we  pursuing  in  the  standardized  reports?  And 
who  is  going  to  ensure  that  a vendor  report  meets  the  standard? 

• In  this  study,  we  have  not  dealt  with  issues  related  to  usage  reporting  in  consortial  arrangements. 
But  as  those  are  becoming  staples  in  research  libraries,  we  need  to  make  efforts  to  make  sure  that 
individual  members  involved  in  consortia  receive  the  same  level  of  usage  statistics  for  their 
institutions  as  in  individual  site  licensing  agreements. 

• More  than  several  organizations  in  library  and  vendor  communities,  national  and  international 
bodies,  are  currently  working  in  this  area  (see  Related  Projects  and  Studies  in  the  Phase  I Report). 
While  these  initiatives  do  not  overlap  exactly  in  terms  of  their  goals  and  scopes,  as  we 
emphasized  in  the  Phase  I report,  there  is  a danger  that  they  may  result  in  conflicting  reporting 
requirements.  Specific  ways  to  coordinate  and  encourage  cooperation  have  yet  to  be  thought  out. 

One  important  accomplishment  during  Phases  I and  II  of  the  project  was  the  initiation  of 
conversations  and  working  together  with  major  database  vendors.  Work  needs  to  continue,  especially  in 
the  standardization  of  key  usage  statistics,  data  delivery,  and  better  documentation  of  definitions  and 
reporting  procedures.  To  that  end,  the  study  team  and  ARL  plan  to  organize  the  second  meeting  with  the 
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involved  vendors  in  conjunction  with  the  ALA  Mid-Winter  meeting  in  January  2002  to  further  move  the 
process  and  make  progress. 

The  accomplishments  achieved  during  Phase  II  of  the  E-Metrics  project  would  not  have  been  possible 
without  the  active  involvement  of  libraries  and  vendors  who  participated  in  the  field-  testing.  We  hope 
that  both  parties  continue  to  be  open-minded  and  to  take  active  roles  working  together  in  the  future. 
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4 DATA  COLLECTION  MANUAL  FOR  THE  RECOMMENDED 
NETWORK  STATISTICS  AND  PERFORMANCE  MEASURES 

This  section  provides  definitions,  data  collection  procedures,  and  discusses  related  issues  pertaining  to 
interpreting  and  using  the  recommended  statistics  and  measures.  The  definitions  and  procedures  were 
derived  from  a month  of  field-testing  at  more  than  a dozen  ARL  libraries.  The  statistics  and  performance 
measures  represent  a minimum  set  of  data  that  need  to  be  collected  continually  and  used.  Individual 
libraries  will  need  to  develop  local  procedures  to  support  data  collection  activities  within  the  guidelines  of 
this  manual.  However,  readers  need  to  recognize  that  the  statistics  and  measures  will  be  refined  and 
extended  continuously  in  the  future. 

4.1  Recommended  Statistics  and  Measures 

Table  4.1  Network  Statistics 


Patron  Accessible  Electronic 
Resources 

R1  Number  of  electronic  full-text  journals 
R2  Number  of  electronic  reference  sources 
R3  Number  of  electronic  books 

Use  of  Networked  Resources  and 
Services 

U 1 Number  of  electronic  reference  transactions 
U2  Number  of  logins  (sessions)  to  electronic  databases 
U3  Number  of  queries  (searches)  in  electronic  databases 
U4  Items  requested  in  electronic  databases 
U5  Virtual  visits  to  library’s  website  and  catalog 

Expenditures  for  Networked 
Resources  and  Related 
Infrastructure 

C 1 Cost  of  electronic  full-text  journals 
C2  Cost  of  electronic  reference  sources 
C3  Cost  of  electronic  books 

C4  Library  expenditures  for  bibliographic  utilities,  networks,  and 
consortia 

C5  External  expenditures  for  bibliographic  utilities,  networks,  and 
consortia 

Library  Digitization  Activities 

D 1 Size  of  library  digital  collection 
D2  Use  of  library  digital  collection 
D3  Cost  of  digital  collection  construction  and  management 

Table  4.2  Performance  Measures 


Performance  Measures 


P 1 Percentage  of  electronic  reference  transactions  of  total  reference 
P2  Percentage  of  virtual  library  visits  of  all  library  visits 
P3  Percentage  of  electronic  books  to  all  monographs 


4.2  Format  of  Data  Collection  Procedures  for  Performance  Statistics  and 
Measures 

The  data  collection  procedures  for  the  statistics  and  performance  measures  are  defined  and  described 
according  to  the  following  criteria: 

Definition:  Describes  each  statistic  or  performance  measure. 
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Rationale:  Discusses  why  the  suggested  statistic  or  performance  measure  is  needed  and/or  how  it  can 
be  useful  to  describe  electronic  resources  and  services. 

Implementation:  Provides  instructions  for  implementing  the  identified  statistic  or  performance 
measure,  categorized  by  collector,  frequency,  procedures,  and  special  considerations,  if  any. 

Collected  by:  Identifies  who  is  responsible  for  collecting  data;  local jefers  to  the  individual  library 
and  vendors  refers  to  the  content  providers  with  whom  the  library  has  contracted  to  provide  electronic 
resources. 

Frequency:  Identifies  how  often  the  statistic/measure  needs  to  be  collected. 

Procedures:  Outlines  the  manner  in  which  the  data  for  a statistic  or  performance  measure  may  be 
collected.  Also  includes  recommendations  for  forms. 

Special  considerations:  Identifies  special  factors  that  need  to  be  considered  during  data  collection  or 
interpretation. 

Related  issues:  Discusses  issues  that  go  beyond  the  suggested  data  collection  procedures,  such  as  the 
availability  of  complementary  statistics,  ways  in  which  statistics  can  be  combined  with  other  statistics, 
and  other  possible  approaches  to  data  collection. 


4.3  Data  Collection  Procedures  for  Performance  Statistics  and  Measures 


The  developed  statistics  for  patron  accessible  resources  account  for  networked  resources  and  services. 
The  current  ARL  membership  criteria  index  lacks  separate  measures  for  electronic  and  networked 
monographs,  serials,  and  bibliographic  utilities.  Though  these  electronic  and  networked  resources  may 
limit  the  amount  of  print  materials  acquired  and  may  cost  more  than  their  print  counterparts,  they  do 
constitute  more  widely  available  resources. 

In  the  electronic  and  networked  realm,  the  more  a library  has,  the  more  materials  are  provided  to 
customers  anytime  and  anywhere.  Although  local  needs  and  available  resource  allocations  may  differ 
from  library  to  library,  the  resource  statistics  allow  academic  research  libraries  to  see  and  to  demonstrate 
to  others  the  changing  nature  of  library  collections  over  the  years.  In  turn,  the  libraries  are  expected  to  use 
them  to  make  decisions  about  resource  allocations  (budget,  staff,  time,  etc.)  and  to  undertake  strategic 
planning  accordingly.  Furthermore,  the  picture  of  available  resources  provides  libraries  with  an 
opportunity  to  offer  valued  services.  However,  because  the  evolving  nature  of  these  statistics  will  rely 
heavily  on  technological  enhancements,  all  libraries  are  encouraged  to  use  extra  caution  while  serving 
their  institutional  goals,  missions,  and  visions. 

R1  — Number  of  Electronic  Full-Text  Journals 

Definition:  Number  of  electronic  full-text  journal  subscriptions  that  the  library  provides  to  users 
either  through  an  individual  institutional  licensing  contract  with  the  provider  of  journals  or  through  other 
arrangements  (e.g.,  regional  or  state  consortium)  for  which  the  library  pays  a reduced  or  no  fee  for  access. 
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The  full-text  journals  should  provide  both  search  and  browse  capabilities  by  title  and  issue.  This  is 
different  from  journal  article  databases,  such  as  Expanded  Academic  ASAP  in  INFOTRAC,  that  do  not 
provide  browsing  capability. 

This  includes  electronic  full-text  journals  offered  by  established  scholarly  journal  publishing  houses 
(e.g.,  Elsevier’s  ScienceDirect  and  Academic  Press’s  IDEAL),  scholarly  societies  (e.g.,  American 
Chemical  Society  journals  and  American  Institute  of  Physics  Online),  and  services  which  aggregate 
content  from  smaller  publishers  or  from  those  publishers  that  prefer  to  use  an  external  delivery  platform 
(Highwire,  OCLC  ECO,  and  EbscoOnline).  This  should  exclude  general-purpose  periodicals  such  as 
magazines  and  newspapers. 

Rationale:  Electronic  access  has  expanded  dramatically  to  provide  a range  of  useful  resources  for 
library  users.  This  statistic  helps  document  the  degree  of  expansion  of  electronic  resource  availability  in 
the  individual  library  and  can  be  used  to  justify  continuation  and  enhancement  of  these  services. 

Research  libraries  act  increasingly  as  gateways  to  a vast  array  of  external  information.  This  measure 
specifically  addresses  the  extensiveness  of  scholarly  content  a library  provides  to  its  user  community.  In 
many  cases,  electronic  access  enables  the  library  to  offer  larger  selections  of  journals  than  it  could  provide 
in  paper  format.  This  statistic  can  also  be  used  for  library  promotion  and  internal  and  external  reporting. 
Particularly,  this  statistic  aims  at  showing  the  changing  nature  of  traditional  scholarly  resources  with 
improved  and  better  access  anytime  and  anywhere. 

Implementation 

• Collected  by:  Local  and  vendors 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly, 
quarterly) 

• Procedures:  It  is  impossible  to  obtain  the  complete  list  of  electronic  full-text  journals  from  a 
single  source.  Possible  sources  for  the  information  include  library  catalog  records  (those  records 
that  point  to  web  addresses),  library  web  pages  that  list  the  journal  titles,  the  internal  electronic 
resource  management  database,  and  vendor  records  (websites  and  contract  documentation). 

1.  Create  a master  list  of  full-text  electronic  journals  from  all  the  sources  available.  Use  a 
spreadsheet  or  database  program  to  organize  and  maintain  the  list. 

2.  Remove  titles  that  do  not  meet  the  above-mentioned  definition  but  keep  duplicate  titles 
resulting  from  multiple  subscriptions  (e.g.,  main  library  and  medical  or  law  libraries). 

3.  Record  the  counts  and  be  sure  they  are  updated  regularly.  The  library  should  also  update  the 
count  information  on  the  library  website  and/or  in  marketing  brochures  on  a regular  basis. 

Special  considerations:  It  is  time-consuming  to  establish  procedures  to  collect  this  statistic  for  the 
first  time.  However,  once  that  is  done,  it  will  be  relatively  easy  to  update  the  information.  (This  applies 
to  other  statistics  and  measures  included  in  the  manual  as  well.) 

Include  journal  titles  that  come  with  print  subscriptions  or  print  plus  online  subscriptions  since  the 
focus  of  the  statistic  has  to  do  with  how  many  scholarly  electronic  journal  titles  users  can  access.  Do  not 
include  free  government  publications  and  free  electronic  journals  to  which  the  library  provides  links. 

Free  government  publications  and  free  electronic  journals  are  a valuable  resource  for  many  libraries. 
How  to  collect  statistics  relating  to  these  resources  will  be  addressed  in  the  future. 
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R2  — Number  of  Electronic  Reference  Sources 

Definition:  Number  of  electronic  reference  sources  and  aggregation  services  that  the  library  provides 
to  users  either  through  an  individual  licensing  contract  with  the  content  providers  or  through  other 
arrangements  (e.g.,  regional  or  state  consortium)  for  which  the  library  pays  a reduced  or  no  fee  for  access. 

This  includes  citation  indexes  and  abstracts;  full-text  reference  sources  (e.g.  encyclopedias,  almanacs, 
biographical  and  statistical  sources,  and  other  quick  fact-finding  sources);  full-text  journal  and  periodical 
article  collection  services  (e.g.,  EBSCOhost,  ProQuest,  Academic  Universe,  and  INFOTRAC  OneFile); 
dissertation  and  conference  proceedings  databases;  and  general-purpose  magazines  and  newspapers. 
Licensed  electronic  resources  also  include  those  databases  that  institutions  mount  locally 

Rationale:  Networking  technology  in  libraries  has  improved  and  increased  dramatically  user  access 
to  a range  of  useful  reference  resources.  This  statistic  documents  the  degree  of  expansion  of  electronic 
resource  availability  and  can  be  used  to  justify  continuation  and  enhancement  of  these  services.  In  the 
1990s,  because  of  the  increasing  popularity  of  the  Internet,  the  ways  reference  interviews  were  held  and 
reference  sources  were  used  changed.  Today,  users  have  electronic  formats  as  well  as  traditional 
reference  sources  to  provide  answers  to  their  reference  questions. 

Research  libraries  traditionally  act  as  gateways  to  a vast  array  of  external  information.  This  measure 
deals  with  the  extensiveness  of  scholarly  content  the  library  provides  to  the  user  community  and  the 
availability  of  reference  sources  on  an  anytime/anywhere  basis.  In  many  cases,  electronic  access  enables 
the  library  to  offer  more  resources  than  it  could  in  paper  format.  This  statistic  can  also  be  used  for  library 
promotion  and  internal  and  external  reporting.  Specifically,  this  statistic  aims  at  showing  the  changing 
nature  of  traditional  scholarly  resources  with  improved  access. 

Implementation 


• Collected  by:  Local  and  vendors 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly, 
quarterly) 

• Procedures:  As  in  the  case  of  the  number  of  electronic  full-text  journals,  it  is  impossible  to  obtain 
the  complete  list  of  databases  from  a single  source.  Possible  sources  for  the  information  include 
library  catalog  records  (those  records  that  point  to  web  addresses),  library  web  pages  that  list  the 
database  titles,  the  internal  electronic  resource  management  database,  and  vendor  records 
(websites  and  contract  documentation). 

1 . Create  a master  list  of  electronic  databases  from  all  the  sources  available.  Use  a spreadsheet 
or  database  program  to  organize  and  maintain  the  list. 

2.  Remove  titles  that  do  not  meet  the  above-mentioned  definition  but  keep  duplicate  titles 
resulting  from  multiple  subscriptions  (e.g.,  main  library  and  medical  or  law  libraries). 

3.  Record  the  counts  and  be  sure  they  are  updated  regularly.  The  library  should  also  update  the 
count  information  on  the  library  website  and/or  in  marketing  brochures  on  a regular  basis. 

Special  considerations:  The  unit  of  measurement  here  is  the  database  not  the  whole  service  provided 
by  a vendor.  For  example,  if  the  library  subscribes  to  OVID  and  the  company  provides  five  databases 
(ABI/Inform,  Books  in  Print,  CINAHL,  INSPEC,  and  PsycINFO),  then  the  count  is  5,  not  1.  By  the  same 
token,  if  the  library  subscribes  to  three  database  packages  (Academic  Universe,  Congressional  Universe, 
and  Statistical  Universe)  from  Lexis-Nexis,  the  count  is  3. 
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This  count  should  not  include  freely  available  databases  to  which  the  library  provides  links  or  library- 
created  finding  aids. 

Freely  available  databases  and  library-created  finding  aids  are  a valuable  resource  for  many  libraries. 
How  to  collect  statistics  relating  to  these  resources  will  be  addressed  in  the  future. 

R3  — Number  of  Electronic  Books 

Definition:  Number  of  electronic  full-text  monographs  that  the  library  offers  to  its  users  either 
through  an  individual  licensing  contract  with  the  content  providers  or  through  other  arrangements  (e.g., 
regional  or  state  consortium)  where  the  library  pays  a reduced  or  no  fee  for  access. 

This  includes  electronic  books  purchased  through  vendors,  such  as  netLibrary  and  Books24x7,  and 
electronic  books  that  come  as  part  of  aggregate  services.  It  excludes  internally  digitized  electronic  books, 
electronic  theses  and  dissertations,  digitally  created  archival  collections  (e.g.,  Early  English  Books 
Online),  and  other  special  collections.  This  also  excludes  publicly  available  electronic  books  to  which  the 
library  provides  web  links.  It  does  not  include  machine-readable  books  distributed  on  CD-ROM,  or 
accompanied  by  print  books. 

Rationale:  Networking  technology  in  libraries  has  improved  and  increased  dramatically  user  access 
to  the  electronic  counterparts  of  some  traditional  sources.  This  statistic  documents  the  degree  of 
expansion  of  e-books.  In  the  mid  90s,  networking  and  resource  sharing  technologies  provided  libraries 
with  print  books  and  e-books  that  were  made  available  through  a library’s  networks. 

Because  the  evolving  nature  of  this  statistic  will  heavily  depend  on  technological  enhancements,  all 
libraries  are  encouraged  to  use  extra  caution  while  pursuing  their  institutional  goals,  missions,  and  visions. 
Moreover,  the  definition  of  e-books  is  still  evolving.  This  statistic  is  an  early  attempt  to  keep  track  of  this 
type  of  resource  as  it  becomes  more  widely  available. 

Implementation 

• Collected  by:  Local  and  vendors 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly, 

quarterly) 

• Procedures: 

1 . For  each  electronic  book  collection,  get  the  electronic  title  counts  from  either  the  providers  or 
catalog  records.  Unlike  electronic  full-text  journals  and  reference  databases,  it  is  not 
necessary  to  list  the  titles  for  each  electronic  book  collection. 

2.  Count  any  duplicate  titles  resulting  from  multiple  subscriptions  (e.g.,  main  library  and 
medical  or  law  libraries). 

3.  Record  the  counts  and  be  sure  they  are  updated  regularly.  The  library  should  also  update  the 
count  information  on  the  library  website  and/or  in  marketing  brochures  on  a regular  basis. 

Special  considerations:  Do  not  include  book  collections  that  are  a part  of  aggregate  services  and 
function  more  as  a reference  collection  (e.g.,  MD  Consult  reference  books,  ProQuest’s  Early  English 
Books  Online,  and  books@OVID).  They  should  be  reported  in  the  electronic  reference  databases. 
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Do  not  include  freely  available  electronic  books  such  as  titles  available  from  the  National  Academy 
Press. 

Related  issues:  Electronic  books,  still  evolving  in  terms  of  technology  and  adoption  for  use,  present  a 
number  of  issues  in  terms  of  definition  and  measurement,  such  as  "location,"  accessibility  (metadata  and 
access  points),  and  use  versus  circulation  (e.g.,  is  on-line  use  for  20  minutes  a circulation,  as  it  would  be 
with  reserve  materials;  or  does  a circulation  of  electronic  books  require  a minimum  period  of  use,  such  as 
24  hours?). 

• What  about  reference  book  collections  provided  by  vendors?  Should  they  be  treated  as  electronic 
books,  for  example,  or  should  they  be  treated  as  a database,  on  the  grounds  that  they  are  used  as 
databases? 

Count  only  those  books  that  a user  can  check  out,  as  they  would  traditional  books.  Unlike  traditional 
books  that  the  library  purchases  and  owns,  electronic  books  can  be  subscribed  to  for  an  ongoing  fee.  In 
this  case,  the  library  accounting  system  may  treat  these  as  serials  rather  than  books  because  of  the  type  of 
payment.  It  is  relatively  easy  to  keep  track  of  the  number  of  electronic  books  right  now  since  most 
libraries  deal  with  only  a handful  of  e-book  vendors,  such  as  netLibrary  and  Books24x7.  But  in  the 
future,  it  will  become  increasingly  difficult  to  do  this  as  the  sources  of  electronic  books  proliferate. 

Finally,  some  provisions  of  contractual  agreements  between  libraries  and  vendors  may  limit  the  level 
of  use  of  e-books.  These  issues  need  be  addressed  in  future  research. 

4.3.2  Statistics  Related  to  Use  of  Electronic  Networked  Resources  and  Services 

High  use  of  a library  resource  or  service  implies  a collection  development  program  that  is  working  to 
create  access  to  the  resources  customers  need.  Use  and  the  need  can  also  identify  resources  and  services 
that  are  seen  as  particularly  valuable  in  the  education  and  research  enterprise  and  should  be  expanded,  or 
perhaps  resources  and  services  that  should  be  discontinued  due  to  lack  of  use  and  interest.  Whether 
provided  by  vendors  or  collected  institutionally,  usage  statistics  can  help  a library  administrator  make 
decisions  and  plan  for  the  future  in  order  to  meet  not  only  users'  expectations  and  needs  but  also 
institutional  goals.  The  reported  data  can  also  provide  other  information  as  to  where  and  when  people  use 
the  library’s  materials  and  how  well  the  library  serves  its  target  audience  and  anticipates  their  potential 
needs. 

The  cost  of  providing  access  to  networked  resources  and  services  can  be  more  expensive  than  that  of 
traditional  counterparts.  Depending  heavily  on  earlier  ICOLC  guidelines,  the  E-Metrics  use  measures  put 
this  in  the  perspective  of  the  changing  academic  research  library  environment.  The  purpose  of  the  use 
measures  is  to  provide  statistics  relating  to  the  use  of  networked  services  and  resources.  Therefore,  it  is 
expected  that  library  administrators  can  reconsider  some  resource  allocation  issues  as  the  number  of 
resources  and  services  tend  to  increase  while  people  are  provided  greater  access.  Please  note  that,  as  with 
most  of  the  statistics  in  this  study,  statistics  related  to  the  use  of  library  resources  and  services  should  be 
revisited  and  perhaps  modified  as  the  technology  advances. 

U1  — Number  of  Electronic  Reference  Transactions 

Definition:  Number  of  electronic  reference  transactions  conducted  via  email,  a library’s  website,  or 
other  network  communications  mechanisms  designed  to  support  electronic  reference.  An  electronic 
reference  transaction  must  include  a question  either  received  electronically  (e.g.,  via  e-mail,  WWW  form, 
etc.)  or  responded  to  electronically.  Those  transactions  that  are  both  received  and  responded  to 
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electronically  are  counted  as  one  transaction.  This  count  excludes  phone  and  fax  traffic  unless  either  the 
question  or  answer  transaction  occurs  via  the  described  manner.  It  includes  the  counts  accrued  from 
participation  in  any  local  and  national  projects,  such  as  DigiRef  and  the  Library  of  Congress’s  CDRS 
(Collaborative  Digital  Reference  Service). 

A reference  transaction  is  an  information  contact,  which  involves  the  knowledge,  use, 
recommendations,  interpretation,  or  instruction  in  the  use  of  one  or  more  information  sources  by  a 
member  of  the  library  staff. 

Rationale:  Libraries  are  making  more  of  their  services  available  electronically  and  are  interested  in 
tracking  the  development  of  a new  and  emerging  library  service.  There  is  a need  to  better  document  this 
transition  to  facilitate  and  improve  resource  allocation  activities.  This  statistic  represents  reference 
activities  conducted  electronically  in  the  library.  It  is  an  attempt  to  measure  reference  transactions 
through  new  electronic  tools. 

Implementation 

• Collected  by:  Local 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (monthly, 

quarterly).  This  statistic  can  be  collected  in  the  same  manner  as  the  library  gathers  other  reference 

transactions  data. 

• Procedures: 

1 . Select  a typical  week  (or  month)  to  run  a sample  study.  Be  sure  to  vary  the  specific  week  (or 
month)  chosen  over  the  course  of  a year  or  from  year  to  year  to  account  for  seasonal 
fluctuations. 

2.  Key  tasks  include  distributing  a daily  tally  sheet,  collecting  the  daily  tally  sheet,  adding  each 
day’s  totals  to  a weekly  figure,  and  being  available  to  respond  to  data  collection  problems 
should  they  occur. 

3.  Transactions  may  be  via  e-mail,  a form  on  a web  page,  etc.  Electronic  reference  transactions 
may  involve  more  than  reference  desk  staff  (e.g.,  web  master,  various  reference  personnel, 
library  director,  volunteers,  etc.).  Establish  an  administrative  procedure  to  report  electronic 
reference  transaction  counts  to  a designated  staff  person,  no  matter  who  receives  the  questions 
or  answers  the  reference  requests. 

4.  Disseminate  the  new  procedure  and  rationale.  Several  notices  throughout  the  year  may  be 
necessary. 

5.  Report  an  electronic  reference  transaction  as  you  would  a face-to-face  reference  transaction. 
Thus,  one  e-mail  request  may  contain  several  reference  questions  taking  varying  times  to 
complete.  For  example,  one  e-mail  request  could  contain  two  relatively  short  reference 
questions  and  one  reference  question  that  took  10-15  minutes  to  answer.  Count  the  number  of 
requests,  not  the  number  of  questions.  Thus,  in  the  example  you  would  report  one  (1)  as  the 
number  of  electronic  reference  transactions  even  though  there  were  three  questions.  Report 
counts  using  pre-established  local  library  reporting  periods  (weekly,  monthly,  etc.). 

6.  Indicate  and  describe  any  additional  methods  used  outside  of  this  definition  and  guidelines. 

Special  considerations:  Unless  the  library  uses  electronic  reference  management  software  to  collect 
and  report  transaction  data,  it  is  difficult  to  keep  track  of  a complete  reference  transaction  cycle  (query 
and  response)  because  of  time-delays  and  the  involvement  of  several  parties. 
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As  stated  in  the  definition,  the  statistic  includes  the  number  of  service  transactions  provided  to  patrons 
outside  the  university  or  the  parent  institution  that  the  library  serves,  through  regional  or  national 
cooperative  efforts  and  through  library  policies. 

Related  issues:  Reference  services  are  undergoing  rapid  changes.  Libraries  are  experimenting  with 
different  modes  of  electronic  reference.  One  could  say  that  simple  email  transactions  that  are  prominently 
mentioned  in  the  procedures  are  not  much  different  from  traditional  reference  services.  How  can  a library 
measure  quality  in  providing  different  types  of  electronic  reference  services  such  as  live-chat  with 
text/voice/video?  Will  this  measure  help  the  library  determine  user  demand  and  thereby  plan  for  resource 
allocation?  To  answer  these  questions,  libraries  need  to  collect  more  detailed  information  such  as  length 
of  time  taken  to  answer  questions,  types  of  questions  by  types  of  transactions,  and  so  on.  Also,  this 
statistic  is  likely  to  produce  some  useful  figures  and  trends  regarding  staff  support  and  allocation  in 
reference  activities. 

U2  — Number  of  Logins  (Sessions)  to  Electronic  Databases 

Definition:  Number  of  user  initiated  sessions  in  licensed  electronic  resources.  A session  or  login  is 
one  cycle  of  user  activities  that  typically  starts  when  a user  connects  to  a database  and  ends  with  explicit 
termination  of  activities  (by  leaving  the  database  through  logout  or  exit)  or  implicit  termination  (time  out 
due  to  user  inactivity).  Licensed  electronic  resources  also  include  those  databases  that  institutions  mount 
locally. 

Rationale:  One  purpose  of  having  a networked  environment  is  to  promote  connectedness  and 
accessibility  to  a variety  of  information  resources,  hence  the  need  for  this  measure.  Also,  the  gradual  shift 
in  the  materials  expenditures  from  traditional  print-  based  resources  to  electronic  databases  can  be 
understood  with  the  measure.  This  measure  will  produce  a count  of  how  often  specific  databases  are  used 
and  complement  traditional  physical  attendance  counts. 

Implementation:  Until  there  is  a standardized  report  generation  system  that  captures  different 
statistics  from  different  content  providers,  we  recommend  that  each  library  develop  an  in-house 
spreadsheet  or  database  to  capture  monthly  usage  statistics  of  licensed  databases.  At  least  on  the  database 
title  level,  usage  statistics  should  be  collected  from  vendors,  entered  into  the  in-house  databank,  and 
maintained  for  reporting  and  analysis. 

• Collected  by:  Vendor 

• Frequency:  Monthly,  but  can  be  reported  quarterly  or  annually 

• Procedures: 

1 . Process  monthly  usage  statistics  from  vendors  and  copy  or  import  the  number  of  attempted 
sessions  in  each  database  (in  each  journal  collection  for  full-text  journals)  to  an  in-house 
spreadsheet  or  database  file. 

2.  Calculate  the  total  sessions  for  a given  month  by  adding  the  number  of  sessions  from  each 
database  or  journal  collection. 

Special  considerations:  Not  all  vendors  report  this  statistic.  Therefore,  it  will  be  necessary  to 
qualify  the  statistic  with  a sentence  such  as  this:  “We  have  150,000  logins  recorded  from  120  databases 
out  of  200  subscribing.  We  cannot  report  this  statistic  for  the  remaining  80  databases  because  the  vendor 
does  not  supply  login  (session)  information  to  customers.” 
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Related  issues:  When  analyzing  the  login  counts,  it  might  be  important  to  explain  any  increases  or 
decreases  in  the  figures.  Specify,  for  example,  whether  the  increase  comes  from  (1)  the  addition  of  new 
databases,  (2)  databases  which  did  not  report  the  statistic  in  the  past  but  have  now  begun  reporting,  (3) 
increased  demand,  and/or  (4)  an  increase  in  the  number  of  simultaneous  users. 

Problems  with  the  comparability  of  login  counts  from  different  vendors  is  a serious  threat  to  the  utility 
of  the  combined  count.  Content  providers  use  different  time-out  thresholds  (ranging  from  7 to  30  minutes 
on  average).  Also,  because  of  the  IP-based  authentication,  several  sessions  conducted  at  the  same  public 
workstation  can  be  counted  as  a single  login.  Alternatively,  libraries  can  collect  attempted  logins  to 
various  licensed  databases  by  making  users  go  through  a central  gateway  (which  counts  all  attempted 
logins).  This  will  ensure  that  one  login  attempt  to  a database  is  the  same  as  a login  to  other  databases. 
However,  what  this  data  collection  method  misses  is  user  logins  that  go  directly  to  content  provider  sites. 
It  is  unclear  how  many  user  logins  fall  into  this  category,  but  the  phenomenon  certainly  results  in  a 
substantial  undercount  of  user  logins. 

While  the  gross  login  figure  is  useful,  it  is  useful  only  for  trend  plotting  and  gross  justification  of 
electronic  resources.  Within  the  library,  the  usage  measures  of  licensed  electronic  resources  have  many 
users  and  uses.  Circulation  of  usage  statistics  on  the  database  title  level  (or  in  an  extreme  case  on  the 
journal  title  level)  and  discussion  of  any  noticeable  changes  (or  lack  thereof)  need  to  occur  at  various 
levels  among  the  concerned  parties,  including  collection  development  personnel,  web  master(s),  technical 
services  staff,  and  so  on. 

U3  --  Number  of  Queries  (Searches)  in  Electronic  Databases 

Definition:  Number  of  user  initiated  queries  (searches)  in  licensed  electronic  resources.  A search  is 
intended  to  represent  a unique  intellectual  inquiry.  Typically,  a search  is  recorded  each  time  a search 
request  is  sent/submitted  to  the  server. 

Rationale:  This  statistic  provides  libraries  with  an  indication  of  the  databases  that  are  most  heavily 
used,  areas  of  user  interest,  database  popularity,  and  a level  of  usage  detail  that  goes  beyond  an  initial 
session.  It  also  can  provide  important  information  for  billing  purposes,  as  some  vendors  charge  for 
database  usage  by  number  of  searches.  This  statistic  can  complement  Ul,  the  number  of  electronic 
reference  transactions,  as  more  user  requests  bypass  staff  mediations.  Some  portion  of  this  statistic  is  also 
analogous  to  in-library  use  of  reference  sources. 

Implementation 

• Collected  by:  Vendor 

• Frequency:  Monthly,  but  can  be  reported  quarterly  or  annually 

• Procedures:  Until  there  is  a standardized  report  generation  system  that  captures  different  statistics 
from  different  content  providers,  we  recommend  that  each  library  develop  an  in-house 
spreadsheet  or  database  to  capture  monthly  usage  statistics  of  licensed  databases.  Usage  statistics 
need  to  be  collected  from  vendors,  entered  into  the  in-house  databank,  and  maintained  for 
reporting  and  analysis. 

1 . Process  monthly  usage  statistics  from  vendors  and  copy  or  import  the  number  of  attempted 
searches  in  each  database  to  an  in-house  spreadsheet  or  database  file. 

2.  Calculate  the  total  number  of  searches  for  a given  month  by  adding  the  number  of  searches 
from  each  database  or  journal  collection. 
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Special  considerations:  Because  some  vendors  do  not  report  this  statistic,  it  will  be  necessary  to 
qualify  the  statistic  with  a sentence  such  as  this:  “We  have  150,000  searches  recorded  from  120  databases 
out  of  200  subscribing.  The  other  80  do  not  provide  this  statistic.” 

Related  issues:  Different  assumptions  about  and  mechanisms  for  collecting  search  counts  by 
different  vendors  are  potential  threats  to  the  combined  count. 

U4  - Number  of  Items  Requested  in  Electronic  Databases 

Definition:  Number  of  items  requested  in  all  of  the  library’s  licensed  electronic  resources.  These 
resources  may  include  journal  articles,  e-books,  reference  materials,  and  non-textual  resources  that  are 
provided  to  the  library's  users  through  licensing  and  contractual  agreements.  The  user  requests  may 
include  viewing,  downloading,  emailing,  and  printing  to  the  extent  the  activity  can  be  recorded  and 
controlled  by  the  server  rather  than  browser. 

The  items  reported  depend  on  the  type  of  content.  Examples  include  citations,  abstracts,  tables  of 
contents,  and  full-text  articles  (ASCII,  HTML,  PDF,  or  PS). 

Rationale:  This  statistic  provides  a circulation  count  for  electronic  contents  in  a way  analogous  to  the 
traditional  circulation  of  books.  Given  the  fact  that  libraries  do  not  have  good  measurements  of  in-house 
materials  usage,  particularly  serials  usage,  this  statistic  helps  libraries  understand  in-library  use  patterns 
that  were  heretofore  difficult  to  measure. 

Implementation 

• Collected  by:  Vendor 

• Frequency:  Monthly,  but  can  be  reported  quarterly  or  annually 

• Procedures:  Until  there  is  a standardized  report  generation  system  that  captures  different  statistics 
from  different  content  providers,  we  recommend  that  each  library  develop  an  in-house 
spreadsheet  or  database  to  capture  monthly  usage  statistics  of  licensed  databases.  Usage  statistics 
should  be  collected  from  vendors,  entered  into  the  in-house  databank,  and  maintained  for 
reporting  and  analysis. 

1 . Process  monthly  usage  statistics  from  vendors  and  copy  or  import  the  number  of  items 
selected  for  viewing,  downloading,  and  emailing  in  each  database.  Count  the  number  and 
type  of  items  users  selected:  abstracts,  citations,  and  full-texts. 

2.  Calculate  the  total  number  of  items  for  a given  month  by  adding  the  number  of  items 
requested  from  each  database  or  journal  collection. 

Special  considerations:  Because  some  vendors  do  not  report  this  statistic,  it  will  be  necessary  to 
qualify  the  statistic  with  a sentence  such  as  this:  “More  than  150,000  items  were  requested  from  120 
databases  out  of  200  subscribing.  The  other  80  do  not  provide  this  statistic.  Among  the  requested  items, 
100,000  were  some  form  of  full-text  records.” 

Related  issues:  Different  vendors  apply  different  assumptions  and  mechanisms  in  collecting  items 
requested  counts.  This  lack  of  standardization  makes  it  difficult  to  calculate  an  aggregate  count. 

We  do  not  have  good  measurement  of  in-house  materials  usage,  particularly  journal  usage.  However, 
electronic  journals  and  databases  allow  libraries  to  find  out  how  often  materials  are  requested.  Having  in- 
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house  usage  figures  is  important  for  understanding  the  dynamics  of  usage  between  print  and  electronic 
journals,  so  that  we  can  ascertain  any  correlation  between  them. 

U5—Virtual  Visits  to  Library's  Website  and  Catalog 

Definition:  This  is  defined  as  user  visits  to  the  library’s  website  or  catalog  from  outside  the  physical 
library  premises  regardless  of  the  number  of  pages  or  elements  viewed.  If  a user  looks  at  16  pages  and  54 
graphic  images  while  at  a website,  that  user  registers  one  visit  on  the  web  server.  All  visits  to  the  website 
should  be  counted  regardless  of  repetition  by  one  user.  A visit  is  usually  determined  by  a user’s  IP 
address,  which  can  be  misleading  due  to  Internet  Service  Providers  (ISPs)  and  Firewalls  or  Proxy  Servers. 
Thus,  this  measure  is  actually  an  estimate  of  the  visits. 

Rationale:  Use  of  the  website  or  catalog  from  outside  the  library  reflects  interest  in  library  services. 
The  role  of  networked  services  is  to  expand  the  reach  of  libraries  beyond  their  physical  boundaries.  This 
statistic  helps  describe  the  significance  of  networked  services  use  by  measuring  the  number  of  virtual 
accesses.  This  will  also  give  an  opportunity  for  the  library  to  compare  the  demand  placed  on  their 
networked  resources  with  that  for  other  popular  information-oriented  websites  (such  as  Excite,  Lycos, 
etc.). 

Implementation: 

• Collected  by:  Local 

• Frequency:  Reported  annually,  although  internal  reporting  will  be  more  frequent  (e.g.,  weekly, 

monthly,  and  quarterly). 

• Procedures: 

1 . Identify  all  sources  of  virtual  visits  to  the  library.  This  may  involve  activities  that  take  place 
on  more  than  one  web  server.  Some  of  the  web  servers  may  be  owned  by  the  library  and 
some  may  be  owned  or  maintained  by  another  department  in  the  university,  an  Internet 
Service  Provider  (ISP),  or  other  library  vendors  (e.g.,  library  OPAC  provider). 

2.  Exclude  internal  use  within  the  premises  of  the  library  from  the  counts  for  this  measure  when 
possible.  Two  common  approaches  are  using  IP  addresses  or  some  form  of  authentication 
tagged  to  each  transaction.  In  terms  of  external  visits  to  the  library,  three  common  sources 
are:  external  access  to  the  library’s  web  page,  remote  logins  (sessions)  to  non-web-based 
library  databases,  and  remotely  accessible  library  OPAC. 

3.  Develop  strategies  for  collecting  the  data  from  each  of  these  sources  of  virtual  visits. 

Different  software  may  be  needed  to  measure  each  electronic  source  of  virtual  visits.  In  some 
cases,  the  library  may  calculate  the  virtual  visits  using  one  or  more  log  analysis  software 
packages.  In  other  cases,  the  external  owner  of  the  web  server  or  service  (the  ISP)  must 
provide  the  data.  Discussions  may  need  to  be  held  with  these  service  providers  to  obtain  the 
needed  data.  In  still  other  cases,  custom  programs  may  have  to  be  developed. 

4.  In  the  case  of  library  web  pages  housed  on  the  library  server,  identify,  configure,  and  install 
appropriate  log  analysis  software.  Determine  log  analysis  software  definition  that 
corresponds  to  the  virtual  visit  definition. 

Note:  Different  log  analysis  software  packages  may  count  virtual  visits  in  different  ways,  so 
the  count  obtained  will  by  necessity  be  an  estimate.  Arrange  with  the  server  technical  staff 
for  regular  (monthly)  reporting  of  internal  visits  at  the  various  user  access  Internet 
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workstations,  external  library  user  virtual  visits,  and  total  virtual  visits  (internal  visits  plus 
external  visits).  Run  the  log  analysis  software. 

5.  In  the  case  of  library  web  pages  housed  on  an  ISP's  server,  identify  the  log  analysis  software 
the  ISP  uses.  Determine  the  definition  of  “visit”  used  by  the  log  analysis  software  that 
corresponds  to  the  virtual  visit  definition  with  the  assistance  of  the  ISP.  Arrange  with  the  ISP 
for  regular  (monthly)  reporting  of  internal  library  visits  at  the  various  user  access  Internet 
workstations,  external  library  user  virtual  visits,  and  total  virtual  visits  (internal  visits  plus 
external  visits). 

6.  Where  virtual  visit  counts  include  the  aggregate  of  internal  and  external  visits,  indicate  this  in 
your  report. 

Special  considerations:  Count  all  visits  to  the  website  regardless  of  repetition  by  one  user  as  long  as 
each  visit  meets  the  criteria  for  this  statistic. 

After  one  user  connects  to  the  Internet,  several  users  could  conduct  multiple  different  searches  in  the 
electronic  service.  In  some  cases,  e.g.,  Internet-accessible  OPAC  use  inside  the  library,  several  users,  one 
after  the  other,  might  make  use  of  the  same  established  connection.  In  most  systems,  a connection  is  cut 
off  after  a specified  period  of  non-use,  thus  solving  part  of  the  problem.  The  best  existing  method  of 
collecting  virtual  visits  is  to  use  log  analysis  software.  The  log  analysis  software  producers  may  define 
virtual  visits  differently.  For  example,  does  a visit  end  after  a time-out  period  of  30  minutes,  15  minutes, 
or  some  other  time?  The  recommended  time-out  period  is  30  minutes,  but  a local  library  may  have  to 
accept  the  available  log  analysis  software's  definition  even  if  it  varies  from  the  above. 

Some  libraries  will  find  it  difficult  to  report  every  virtual  visit.  For  example,  libraries  may  have 
difficulty  counting  the  use  of  library  OPACS  because  their  vendors  do  not  provide  this  information. 

Make  a record  of  those  sources  of  virtual  visits  not  counted.  Do  not  estimate  virtual  visits  for  which  data 
are  not  available. 

Related  issues:  This  measurement  requires  a relatively  high  degree  of  technical  skills  either  on  staff 
or  available  from  the  library’s  website  host. 

4.3.3  Statistics  Related  to  Expenditures  for  Electronic  Resources  and  Related 

Infrastructure 

This  portion  of  the  statistics  is  based  on  the  ARL  Supplementary  Statistics  Survey  (the  most  recent 
survey  instruction  available  at  http://www.arl.Org/stats/arlstat/#sup).  In  collecting  the  statistics,  the  library 
should  refer  to  the  procedures  followed  and  the  amounts  reported  in  response  to  the  ARL  Supplementary 
Statistics  Survey. 

These  statistics  were  developed  by  ARL  to  determine  expenditure  patterns  on  electronic  and 
networked  resources  and  the  effect  of  new  types  of  library  resources  and  services,  those  delivered  both 
individually  and  collectively  with  other  institutions,  on  library  expenditures.  These  measures  are  expected 
to  help  ARL  libraries  justify  their  growing  budgets  due  to  the  great  expense  of  electronic  and  networked 
services.  These  measures  can  help  answer  such  questions  as:  How  much  are  research  libraries  spending 
for  electronic  resources  collectively  and  how  much  on  average?  How  do  expenditures  for  electronic 
resources  compare  across  several  research  libraries? 
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We  have  not  included  the  cost  of  the  technical  staff  and  their  training,  the  networking  and  equipment 
to  provide  access  to  the  electronic  resources  as  well  as  the  time  of  all  the  staff  involved.  This  will  have  to 
be  addressed  in  the  future. 

General  Introduction  to  C1-C3 


The  report  should  include  expenditures  for  electronic  indexes  and  reference  tools,  electronic  full-text 
periodical  collections  and  electronic  journal  back-files,  and  online  searches  of  remote  databases  - whether 
accessed  remotely  or  installed  locally  from  CD-ROM,  magnetic  tapes,  etc.  The  report  should  also  include 
expenditures  for  materials  purchased  jointly  with  other  institutions  if  such  expenditures  can  be  separated 
from  other  charges  for  joint  services,  fees  paid  to  bibliographic  utilities  if  the  portion  paid  for  computer 
files  and  search  services  can  be  separately  counted,  and  equipment  costs  when  they  are  inseparably 
bundled  into  the  price  of  the  information  product. 

Expenditures  for  bibliographic  utilities,  networks,  and  consortia  that  are  unrelated  to  end-user 
database  access  should  be  reported  in  C4,  not  in  Cl  through  C3. 

Cl  — Cost  of  Electronic  Full-Text  Journals 

Definition:  Expenditures  for  electronic  full-text  journal  subscriptions  that  the  library  provides  to  its 
users.  Include  both  initial  purchase  cost,  membership  fees  (such  as  JSTOR)  as  well  as  annual  access  and 
service  fees  paid  directly  or  through  consortia  arrangements. 

Rationale:  This  statistic,  cost  of  electronic  full-text  journals,  was  developed  by  ARL  to  find  out  how 
much  libraries  are  spending  on  electronic  full-text  journals  and  how  new  forms  of  electronic  journals  are 
replacing  traditional  journals  and  scholarly  publications.  It  also  indicates  the  extent  of  budget  allocations 
for  electronic  resources.  Furthermore,  this  statistic  allows  libraries  to  calculate  unit  costs  of  e-joumals 
after  collecting  Cl  and  R1  statistics,  and  thus  aids  libraries  in  deciding  how  effectively  they  are  serving 
their  potential  and  intended  audiences,  and  in  benchmarking  with  the  other  institutions. 

Implementation 

• Collected  by:  Local 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly, 
quarterly) 

• Procedures:  For  the  definition  of  electronic  full-text  journals,  please  refer  to  the  definition  of  R1 . 
Current  library  accounting  systems  do  not  support  coding  of  materials  expenditures  by  the 
categories  used  in  the  manual.  Therefore,  it  may  become  necessary  to  create  an  in-house 
spreadsheet  or  database  file  to  keep  track  of  cost  information  according  to  the  types  of  resources 
(e-joumals,  reference  databases,  and  e-books).  Preferably  a single  file  will  contain  contract 
information  (duration,  cost),  updated  title  counts,  and  reported  usage  statistics.  Significant 
coordination  is  required  for  setting  up  the  structure  of  the  file,  but  in  the  long  run  may  streamline 
many  aspects  of  the  management  of  electronic  licensed  materials. 

1 . Gather  reports  and  invoices  related  to  electronic  databases  and  resources  for  the  reporting 
period.  These  documents  are  typically  handled  by  the  library’s  accounting  office. 

2.  If  you  have  not  done  so,  organize  the  data  using  the  sample  worksheet  in  Appendix  C,  Figure 

C.l. 
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3.  Even  if  a licensing  contract  or  consortium  arrangement  period  is  different  from  the  reporting 
period,  use  the  annual  licensing  fee  to  calculate  the  statistics. 

4.  If  a fee  is  paid  to  a consortium  or  other  joint  arrangement,  include  the  amount.  In  the  case 
where  a fee  is  paid  for  an  aggregate  service  and  the  service  contains  different  categories  of 
resources  (full-text  journals  and  reference  sources)  as  a bundle,  use  an  estimate  based  on 
expected  or  historical  use,  or  list  prices. 

5.  Note  any  major  commitments  (such  as  JSTOR  one-time  costs)  that  do  not  occur  year  to  year 
and  that  significantly  influence  the  reported  amount. 

Special  considerations:  Whereas  the  patron  accessible  resource  counts  reflect  the  extensiveness  of 
electronic  resources  at  a given  point  in  time  (most  likely  at  the  end  of  the  reporting  period,  be  it  a month 
or  a year),  the  cost  figures  cover  a period  of  time.  Ideally  the  amount  of  money  spent  reflects  the  number 
and  extensiveness  of  resources.  Prorating  licensing  fees  addresses  part  of  the  problem  of  matching  the 
resources  with  the  money  spent,  but  can  be  very  time  consuming.  You  can  report  the  annual  amount  paid 
without  prorating  on  the  basis  that  over  the  years  the  figures  will  even  out. 

Some  electronic  full-text  journals  come  either  as  a free  service  with  a print  subscription  or  as  part  of  a 
print-plus-online-access  subscription  (the  library  pays  extra  for  electronic  access).  In  the  first  case,  the 
problem  is  whether  or  not  to  post  any  amount  for  the  cost  of  electronic  access.  In  the  latter  case,  the 
question  is  how  much  of  the  cost  can  be  attributed  to  electronic  access. 

C2  --  Cost  of  Electronic  Reference  Sources 

Definition:  Expenditures  for  electronic  reference  sources  and  aggregate  services  that  the  library 
provides  to  users  either  through  individual  licensing  contracts  with  content  providers  or  through  consortia 
or  other  arrangements  where  the  library  pays  some  fees.  These  fees  include  both  annual  access  fees  and 
other  service  costs  paid  to  the  vendor  directly  or  through  consortial  arrangements. 

Rationale:  This  statistic,  cost  of  electronic  reference  sources,  was  developed  by  ARL  to  determine 
how  much  libraries  are  spending  on  electronic  reference  sources  and  how  new  forms  of  electronic 
reference  sources  are  replacing  traditional  reference  materials.  It  also  gives  insight  into  shifts  in  budget 
allocations  from  print  to  electronic  materials,  or  new  allocations  exclusively  for  electronic  materials. 
Furthermore,  this  statistic  allows  libraries  to  calculate  unit  costs  of  electronic  reference  sources  after 
collecting  C2  and  R2  figures.  This  figure  assists  libraries  in  making  decisions  about  how  effectively  they 
are  serving  their  potential  and  intended  audience,  and  in  benchmarking  with  other  institutions. 

Implementation 

• Collected  by:  Local 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly, 
quarterly) 

• Procedures:  For  the  definition  of  electronic  reference  sources,  please  refer  to  the  definition  of  R2. 
For  libraries  that  do  not  have  acquisitions  systems  which  support  coding  of  materials  expenditures 
by  the  categories  used  in  the  manual,  it  may  be  necessary  to  create  an  in-house  spreadsheet  or 
database  file  to  keep  track  of  cost  information  according  to  the  types  of  resources  (e-joumals, 
reference  databases,  and  e-books).  Preferably  a single  file  will  contain  contract  information 
(duration,  cost),  updated  title  counts,  and  reported  usage  statistics. 
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1.  Gather  reports  and  invoices  related  to  electronic  databases  and  resources  for  the  reporting 
period.  These  documents  are  typically  handled  by  the  library’s  accounting  office. 

2.  If  you  have  not  done  so,  organize  the  data  using  the  sample  worksheet  in  Appendix  C,  Figure 
C.l. 

3.  Even  if  a licensing  contract  or  consortium  arrangement  period  is  different  from  the  reporting 
period,  use  the  annual  licensing  fee  to  calculate  the  statistics. 

4.  If  a fee  is  paid  to  a consortium  or  through  other  joint  arrangement,  include  the  amount.  If  a 
fee  is  paid  for  an  aggregate  service  and  the  service  contains  different  categories  of  resources 
(full-text  journals  and  reference  sources)  as  a bundle,  use  an  estimate  based  on  expected  or 
historical  use,  or  list  prices. 

5.  In  the  comments  field  of  the  sample  worksheet  (Appendix  C,  Figure  C.l),  report  any  major 
commitments  that  do  not  occur  year  to  year  and  that  significantly  influence  the  reported 
amount. 

Special  considerations:  Whereas  the  patron  accessible  resource  counts  reflect  the  extensiveness  of 
electronic  resources  at  a given  point  in  time  (most  likely  at  the  end  of  reporting  period,  be  it  a month  or  a 
year),  the  cost  figures  cover  a period  of  time.  Ideally  the  amount  of  money  spent  reflects  the  number  and 
extensiveness  of  resources.  Prorating  licensing  fees  addresses  part  of  the  problem  of  matching  the 
resources  with  the  money  spent,  but  can  be  very  time  consuming.  You  can  report  the  annual  amount  paid 
without  prorating  on  the  basis  that  over  the  years  the  figures  will  even  out. 

C3  — Cost  of  Electronic  Books 

Definition:  Expenditures  for  electronic  full-text  monographs  that  the  library  offers  to  its  users. 
Include  both  initial  purchase  costs  and  membership  fees  as  well  as  annual  access  and  service  fees  paid 
directly  or  through  consortia  arrangements. 

Rationale:  This  statistic,  cost  of  electronic  books,  was  developed  by  ARL  to  determine  how  much 
libraries  were  spending  on  electronic  books.  It  also  gives  an  idea  about  the  extent  of  budget  allocations 
for  electronic  resources.  Furthermore,  this  statistic  allows  libraries  to  calculate  unit  costs  of  e-books  after 
collecting  C3  and  R3  statistics,  aids  them  in  determining  how  effectively  they  are  serving  their  potential 
and  intended  audiences,  and  assists  them  in  benchmarking  with  other  institutions. 

Implementation 

• Collected  by:  Local 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (monthly, 
quarterly) 

• Procedures:  For  the  definition  of  electronic  books,  please  refer  to  the  definition  of  R3.  Current 
library  accounting  systems  generally  do  not  support  coding  of  materials  expenditure  by  the 
categories  used  in  the  manual.  [See  note  for  3.2.4.5]  Therefore,  it  may  become  necessary  to 
create  an  in-house  spreadsheet  or  database  file  to  keep  track  of  cost  information  according  to  the 
types  of  resources  (e-joumals,  reference  databases,  and  e-books).  Preferably  a single  file  will 
contain  contract  information  (duration,  cost),  updated  title  counts,  and  reported  usage  statistics. 

1 . Gather  reports  and  invoices  related  to  electronic  databases  and  resources  for  the  reporting 
period.  These  documents  are  typically  handled  by  the  library’s  accounting  office.  You  may 
also  need  to  review  circulation  records  to  verify  the  accuracy  of  invoices  if  additional  per-use 
fees  are  paid  (royalty  on  use,  as  with  E-reserves). 
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2.  If  you  have  not  done  so,  organize  the  data  using  the  sample  worksheet  in  Appendix  C,  Figure 
C.l. 

3.  Even  if  a licensing  contract  or  consortium  arrangement  period  is  different  from  the  reporting 
period,  use  the  annual  licensing  fee  to  calculate  the  statistics. 

4.  If  a fee  is  paid  to  a consortium  or  other  joint  arrangement,  include  the  amount. 

5.  Note  any  major  commitments  (such  as  netLibrary  purchase  costs)  that  do  not  occur  year  to 
year  and  that  significantly  influence  the  reported  amount. 

Special  considerations:  Whereas  the  patron  accessible  resource  counts  reflect  the  extensiveness  of 
electronic  resources  at  a given  point  in  time  (most  likely  at  the  end  of  reporting  period,  be  it  a month  or  a 
year),  the  cost  figures  cover  a period  of  time.  Ideally  the  amount  of  money  spent  reflects  the  number  and 
extensiveness  of  resources.  Prorating  licensing  fees  addresses  part  of  the  problem  of  matching  the 
resources  with  the  money  spent,  but  it  can  be  very  time  consuming.  You  can  report  the  annual  amount 
paid  without  prorating  with  the  rationale  that  over  the  years  the  figures  will  even  out. 

Traditionally  books  are  purchased  on  a one-time  payment  in  exchange  for  permanent  ownership  by 
the  library.  However,  with  regard  to  electronic  books,  it  appears  that  some  arrangements  allow  libraries  to 
subscribe  to  an  e-book  collection  at  a predetermined  fee  and  for  a predetermined  interval  of  time.  We  are 
concerned  with  the  format  of  the  material,  not  the  subscription  or  payment  arrangement.  These  materials 
should  be  counted  as  books,  not  serial  publications. 


Related  issues:  In  many  instances,  the  physical  form  of  the  material  (print,  electronic)  may  change 
the  nature  of  the  object.  An  electronic  book  is  a good  example.  With  enhancements  such  as  full-text 
searching  (although  print  books  too  have  some  search  capability  through  tables  of  contents  and  indexes), 
electronic  books  support  new  forms  of  searching  not  present  in  print. 


C4  — Library  Expenditures  for  Bibliographic  Utilities,  Networks,  and  Consortia* 


Definition:  Expenditures  paid  by  the  library  for  services  provided  by  national,  regional,  and  local 
bibliographic  utilities,  networks,  and  consortia  such  as  OCLC,  RLG,  excluding  fees  paid  for  user  database 
access  and  subscriptions,  which  should  be  reported  in  Cl  through  C3. 


Rationale:  This  statistic  is  based  on  the  ARL  Supplementary  Statistics  Survey.  It  was  developed  by 
ARL  to  determine  how  much  money  libraries  spend  for  bibliographic  utilities,  networks,  and  consortia. 
Because  individual  libraries  often  have  to  deal  with  special  provisions  and  funding  issues  related  to 
contracts,  this  statistic  may  not  lend  itself  to  comparability  among  ARL  member  libraries.  Nevertheless,  it 
represents  an  attempt  to  keep  track  of  the  financial  relationships  between  bibliographic  utilities  and 
libraries.  Although  this  may  provide  very  limited  comparability,  it  is  an  estimate  of  the  cost  of 
bibliographic  utilities,  networks,  and  consortia. 


Implementation 

• Collected  by:  Local 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly, 
quarterly) 


* This  corresponds  to  item  3a  in  the  ARL  Supplementary  Statistics 
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• Procedures: 

1.  Gather  reports  and  invoices  with  bibliographic  utilities,  networks,  and  consortia  of  which  the 
library  is  a member  for  the  whole  or  part  of  the  reporting  period.  These  documents  are 
typically  handled  by  the  library’s  accounting  office. 

2.  Identify  only  those  expenditures  paid  to  the  bibliographic  utilities,  networks,  and  consortia  for 
membership,  maintenance,  and  other  infrastructure.  Do  not  include  expenditures  that  are 
directly  attributable  to  access  of  electronic  resources.  Those  expenditures  should  be  included 
in  Cl  through  C3. 

For  instance,  if  your  library  paid  a total  of  $100,000  to  OCLC  for  its  various  services  and 
your  best  guess  of  electronic  database  access  portion  of  the  services  is  80%,  then  you  should 
report  $80,000  for  C2  and  the  remaining  $20,000  for  C4. 

3.  Even  if  a membership  or  consortium  period  is  different  from  the  reporting  period,  use  the 
amount  of  the  membership  or  consortium  agreement. 

4.  Use  the  sample  form  in  Appendix  C,  Figure  C.2  to  compile  the  expenditures. 

Special  considerations:  Prorating  can  be  time  consuming.  Consortia  or  other  memberships  may 
bring  additional  benefits,  such  as  subscriptions,  training  or  preferential  pricing  for  acquisition  of  materials. 
It  may  be  difficult  to  separate  pure  membership  fees  from  value-added  services  of  membership  (e.g., 
original  catalog  credits  from  OCLC  that  may  be  used  to  offset  costs  of  databases,  purchase  of  catalog 
records,  etc.).  Report  the  annual  amount  paid  without  prorating  with  the  rationale  that  over  the  years  the 
figures  will  even  out. 

C5  — External  Expenditures  for  Bibliographic  Utilities,  Networks,  and  Consortia* 

Definition:  Expenditures  paid  by  external  agencies,  such  as  state  government  agencies,  on  the 
library’s  behalf  for  access  to  computer  files,  electronic  serials,  or  search  services  through  a centrally 
funded  system  or  consortial  arrangements.  Examples  include  state-  (or  province-)  supported  networks 
such  as  VIVA  (Virginia),  CNSLP  (Canadian  National  Site  Licensing  Project),  and  the  University  of 
California’s  California  Digital  Library  Expenditure. 

Rationale:  Like  statistic  C4,  this  statistic  is  based  on  the  ARL  Supplementary  Statistics  Survey.  It 
was  developed  by  ARL  to  determine  how  much  money  is  spent  for  bibliographic  utilities,  networks,  and 
consortia  on  libraries'  behalf  for  access  to  computer  files,  serials,  and/or  services  through  consortial 
arrangements.  Because  of  contractual  issues,  this  statistic  may  provide  little  comparability  among  ARL 
member  libraries.  Nevertheless,  it  can  give  ARL  members  an  estimate  of  the  external  costs  of 
bibliographic  utilities,  networks,  and  consortia. 

Implementation 

• Collected  by:  Local  and  external  bodies  such  as  regional  and  academic  consortia 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly, 

quarterly). 


* This  corresponds  to  item  3b  in  the  ARL  Supplementary  Statistics 
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• Procedures: 


1 . Gather  reports  and  invoices  with  bibliographic  utilities,  networks,  and  consortia  that  are 
related  to  electronic  databases  and  resources  for  the  reporting  period.  These  documents  are 
typically  handled  by  the  library’s  accounting  office.  However,  they  can  be  maintained 
outside  the  organization  and,  in  some  instances,  may  only  be  provided  to  libraries  upon 
demand. 

2.  Find  out  how  much  of  the  central  funding  is  attributable  to  your  library.  For  example,  if  your 
library  contributes  a total  of  $60,000  over  a period  of  three  years  to  a state  consortium  that  has 
a matching  contribution  of  $ 120,000  for  the  same  period,  the  amount  to  report  as  C5  for  a 
given  year  during  the  three-year  period  will  be  $40,000  ($120,000  x 1/3).  The  library’s 
contribution  ($60,000)  has  to  be  divided  annually  and  posted  in  Cl  through  C3. 

If  the  specific  dollar  amount  is  not  known,  but  the  total  student  FTE  for  the  consortium  and 
the  amount  spent  for  the  academic  members  are  known,  divide  the  overall  amount  spent  by 
your  institution’s  share  of  the  total  student  FTE.  Alternatively,  if  the  consortium  is  comprised 
of  different  types  of  institutions  (academic,  public,  or  corporate),  but  the  library  has 
information  about  the  portion  of  its  own  use  among  the  consortium  participants,  multiply  the 
total  amount  by  the  percentage  of  known  (or  estimated)  usage  rate. 

3.  Asa  last  resort,  consult  with  a staff  member  overseeing  the  consortium  or  the  central  funding 
system  to  get  an  estimate  of  the  portion  of  the  central  funding  that  is  attributable  to  the  library. 
Please  make  a note  of  this  in  the  comments  field  in  the  sample  worksheet  (Appendix  C, 

Figure  C.3). 

4.  Use  the  sample  form  in  Appendix  C Figure  C.3  to  compile  the  expenditures. 


Comprised  of  resource  and  use  measures,  the  digital  collection  measures  attempt  to  describe  where 
libraries  are  in  creating  and  making  available  local  (perhaps  unique)  content  that  may  not  have  been 
previously  accessible.  Such  collections  can  attract  students  and  faculty  to  your  university  and  thereby 
enhance  the  institution’s  reputation.  As  more  libraries  digitize  resources,  more  users  will  be  able  to 
retrieve  those  unique  resources  at  anytime  and  from  anywhere.  Digital  library  projects,  as  well  as  other 
network  resources  and  services,  also  will  serve  increasing  numbers  of  students  taking  courses  online. 

Collecting  library  digitization  measures  may  provide  an  opportunity  for  benchmarking  and  may 
encourage  libraries  to  devote  more  time  and  allocate  more  resources  to  this  worthwhile  endeavor.  It 
should  be  noted  that  these  statistics  represent  a very  early  attempt  to  measure  digitization  of  resources;  as 
time  passes  and  the  technology  advances,  some  of  the  definitions  and  procedures  may  need  to  be  revisited 
and  modified.  During  the  field-testing  it  was  reported  that  storing  and  maintaining  digitized  resources  had 
been  an  issue.  The  unavailability  of  an  appropriate  infrastructure  in  some  institutions  meant  that  the 
project  did  not  include  statistics  related  to  library  digitization  projects. 

Libraries  archive  the  scholarly  output  of  their  institutions  - theses  and  dissertations  - in  both  paper 
and  digital  form.  Digital  collections  also  provide  new  opportunities  with  faculty  to  archive  research 
results.  These  statistics,  although  preliminary,  form  a basis  for  tracking  these  issues. 


ARL  E-Metrics  Phase  II  Report  73  October  200 1 


4.3.4  Statistics  Related  to  Library  Digitization  Activities 


Measures  and  Statistics  for  Research  Library  Networked  Services 


D1  — Size  of  Library  Digital  Collection 

Definition:  Library  digital  collection  refers  to  digital  materials  (texts,  images,  and  audio-visuals) 
created  in  or  converted  from  different  formats  (e.g.,  paper,  microfilm,  tapes,  etc.)  by  the  library  and  made 
available  to  users  electronically.  This  includes  electronic  theses  and  dissertations  (ETDs),  special 
collections  materials,  maps,  sound  recordings,  films,  and  other  digital  materials  that  are  not  purchased  or 
acquired  from  outside  through  individual  or  consortial  licensing  agreements.  Includes  the  number  of  titles 
and  size  (in  gigabytes)  by  sub-categories  (ETD,  visual  materials,  texts,  multimedia),  and  as  an  aggregate 
at  the  end  of  the  reporting  period.  Also  includes  the  number  of  items  (titles)  added  during  the  reporting 
period. 

The  types  of  formats  in  Appendix  C,  Figure  C.4,  refer  to  original  formats  rather  than  the  digitized 
outputs.  Examples  of  visual  materials  include  photos,  maps,  and  postcards.  Examples  of  text  include 
books,  journal  articles  and  pamphlets.  Examples  of  multimedia  include  audio,  video,  and  other  interactive 
materials.  However,  this  statistic  does  not  include  any  back  up  copies  or  mirror  sites  because  items 
should  be  counted  only  once. 

Rationale:  Collecting  library  digitization  measures  may  provide  an  opportunity  for  benchmarking  in 
terms  of  file  sizes  for  the  resources  that  have  been  digitized.  Moreover,  the  statistic  can  demonstrate  that 
libraries  are  not  merely  brokers  of  external  information  resources,  but  also  producers  of  information 
content  and  useful  finding  aids. 

This  statistic  provides  information  on  the  extent  of  digital  library  projects,  the  life  cycle  of  such 
projects,  and  the  "virtual  space"  requirements  of  such  collections. 

Implementation 

• Collected  by:  Local 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly, 
quarterly) 

• Procedures:  For  cases  in  which  multiple  digital  formats  (derivatives)  were  produced  from  an  item, 
count  it  only  once  based  on  the  type  of  item  that  was  digitized.  For  example,  if  a 100-page  book 
was  digitized  in  100  TIFF  files,  each  containing  a page,  a 100-page  PDF  file,  and  10  PDF  files 
(one  PDF  file  for  each  of  10  chapters),  count  it  as  a single  text  with  100  pages.  If  a derivative 
item  was  used  as  the  source,  do  not  count  the  outputs.  But  in  the  total  size  (in  gigabytes)  include 
all  versions  of  derivatives. 

1 . Designate  a staff  member  to  coordinate  the  collection  of  this  statistic.  The  person  should  be 
well  aware  of  library  digital  collection  activities. 

2.  Identify  library  staff  in  charge  of  various  digital  library  projects  and  initiatives. 

3.  It  is  necessary  to  conduct  an  inventory  of  digital  material  stock  using  the  sample  tally 
worksheet  in  Appendix  C,  Figure  C.6  if  it  has  not  been  done  already.  If  this  inventory 
information  is  already  available,  enter  it  into  the  worksheet.  When  the  inventory  is 
completed,  summarize  the  information  using  the  sample  worksheet  in  Appendix  C,  Figure 
C.4.  Add  additional  categories  if  necessary. 

4.  After  obtaining  the  inventory  information,  ask  staff  members  to  keep  track  of  additional 
output  regularly  using  the  sample  tally  worksheet  in  Appendix  C,  Figure  C.6. 
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5.  At  the  end  of  the  reporting  period,  collect  the  worksheets  and  calculate  the  total  production 
during  the  reporting  period  using  the  worksheet  in  Appendix  C,  Figure  C.5.  Add  additional 
categories  if  necessary. 

Related  issues:  Realistically,  each  digital  collection  is  unique  in  terms  of  the  production  process,  the 
way  it  is  intended  to  be  used,  its  focus,  and  maintenance.  It  is  important  to  use  appropriate  units  of 
measurement  to  describe  the  overall  size  and  extensiveness  of  the  whole  collection. 

Because  of  the  wide  variations  of  the  types  and  features  of  digital  collections  constructed  at  ARL 
institutions,  this  statistic  may  be  more  useful  locally  than  for  comparison  across  ARL  member  libraries. 
Benchmarking  may,  however,  be  possible  from  the  data  collected  to  produce  some  qualitative  and 
quantitative  indicators  as  to  the  extent  of  digital  library  collection  activities  and  different  emphases  across 
the  ARL  membership. 

D2  — Use  of  Library  Digital  Collection 

Definition:  Number  of  times  library  digital  collection  titles  and  physical  files  were  accessed  and  the 
number  of  searches  (queries)  conducted  (if  there  is  such  a capability)  during  the  reporting  period. 

Rationale:  Each  digital  collection  is  unique  in  terms  of  its  focus,  production  process,  and  the  ways  it 
is  intended  to  be  used  and  maintained.  Therefore,  because  of  the  wide  variations  of  the  types  and  features 
of  these  library  collections  constructed  at  different  ARL  institutions,  this  statistic  needs  to  be  collected  and 
used  locally  instead  of  across  ARL  member  libraries.  Nevertheless,  this  statistic  has  the  potential  to 
produce  some  qualitative  and  quantitative  indicators  as  to  how  these  collections  are  being  used  and 
serving  the  intended  user  community's  needs. 

Implementation 

• Collected  by:  Local 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly, 

quarterly) 

• Procedures: 

1.  Designate  a staff  member  to  coordinate  the  collection  of  this  statistic.  The  person  should  be 
well  versed  in  the  use  of  web  log  software  and/or  statistics  provided  by  the  software.  This 
person  will  act  as  a liaison  for  staff  members  who  are  responsible  for  managing  library  digital 
collections.  Obtaining  the  statistic  may  require  some  level  of  programming  (e.g.,  Unix 
scripting  and  SQL). 

2.  Items  accessed  can  be  collected  in  various  ways,  and  depending  on  your  library’s 
environment,  your  library  may  need  to  collect  different  access  statistics. 

Although  you  are  asked  to  collect  both  title  access  and  physical  file  access,  if  it  takes  too 
much  time  and  effort  to  collect  the  title  access,  report  the  physical  file  access  count  only.  For 
example,  a book  can  be  digitized  and  made  into  10  PDF  files,  each  containing  a chapter,  for 
access.  Suppose  a user  viewed  five  PDF  files  out  of  10.  In  this  case,  you  will  have  five 
physical  item  accesses  and  one  title  access.  Usually  it  is  easier  to  have  a physical  item  access 
count,  while  it  takes  custom  programming  to  compute  the  title  access  count  as  most  off-the- 
shelf  web  traffic  software  packages  do  not  provide  this. 
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Do  not  report  web  page  hits.  Instead,  count  how  many  times  the  digitized  items  were 
accessed  (the  exact  name  for  item  access  may  vary  depending  on  the  type  of  web  traffic 
analysis  software  being  used  in  the  library). 

If  a search  capability  is  a feature  of  a library  digital  collection,  the  total  number  of  searches 
submitted  needs  to  be  collected.  A search  represents  an  explicit  user  request  for  specific 
information  in  a database  and  is  expressed  usually  in  the  form  of  word  strings.  Clicks  on  web 
page  buttons,  such  as  “Next”  and  “Previous,”  do  not  count  as  user  searches. 

You  might  want  to  install  web  traffic  analysis  software  (e.g.,  WebTrends,  Web  Tracks)  on  the 
library  web  servers  housing  library  digital  collection  materials,  if  the  web  servers  do  not  have 
such  software  already.  You  might  want  to  consider  installing  a trial  version  that  gives 
between  30-180  days  of  free  trial. 

Read  the  description  of  reported  statistics  carefully  and  make  sure  that  the  software  provides 
what  you  want. 

3.  If  continuous  collection  of  use  statistics  is  not  possible  or  desirable,  select  a typical  week  (or 
month)  to  run  a sample  study.  Be  sure  to  vary  the  specific  week  (or  month)  chosen  over  the 
course  of  a year  or  from  year  to  year  to  account  for  seasonal  fluctuations.  Extrapolate  based 
on  the  sample  data. 

4.  At  the  end  of  the  report  period,  use  the  log  analysis  report  to  calculate  the  number  of  accesses 
to  library  digital  collection  items.  Use  the  sample  report  in  Appendix  C,  Figure  C.7,  to 
organize  the  information. 

Special  considerations:  To  the  extent  possible,  exclude  accesses  by  web  search  spiders.  Also,  do  not 
include  accesses  to  auxiliary  (or  incidental)  items  that  are  not  part  of  the  library  digital  collection  content 
(.gif  buttons  and  image  maps  for  navigation).  Note  the  method  used  and  include  a description  of  any 
filtering  done. 

Related  issues:  This  statistic  needs  to  be  collected  and  used  locally  instead  of  across  ARL  member 
libraries  because  of  the  wide  variations  of  the  types  and  features  of  digital  collections  constructed  at  ARL 
institutions. 

D3  - Cost  of  Digital  Collection  Construction  and  Management 

Definition:  Annual  direct  costs  (personnel,  equipment,  software,  contracted  services  and  similar 
items)  spent  to  create  digital  materials  (texts,  images,  and  multimedia)  or  to  convert  existing  materials  into 
digital  form  for  the  purpose  of  making  them  electronically  available  to  users.  Include  expenditures  related 
to  digitization,  OCR,  editorial,  creation  of  markup  texts,  preparation  of  metadata  for  access  to  digitized 
materials,  data  storage,  and  copyright  clearance.  Exclude  expenditures  for  information  resources 
purchased  or  acquired  from  outside  the  institution  through  individual  or  consortial  licensing  agreements. 

Rationale:  The  cost  of  each  digital  collection  construction  may  vary  significantly,  depending  on  the 
size  of  the  collection,  conditions  of  the  sources  before  digitizing,  available  infrastructure,  staff  allocation, 
timeline,  and  administrative  support.  This  statistic  should  be  collected  and  used  locally  instead  of  across 
ARL  member  libraries  because  of  the  wide  variability  among  these  library  collections  constructed  at 
different  ARL  institutions.  Nevertheless,  this  statistic  has  the  potential  to  provide  quantitative  indicators 
as  to  how  costly  these  efforts  are,  how  much  resource  allocation  (i.e.,  budget  allocation,  staffing, 
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infrastructure,  etc.)  is  needed,  and  how  well  they  serve  the  intended  user  community’s  needs  (e.g.,  to 
account  for  internal  and  external  costs  to  construct  and  manage  digital  collections  at  ARL  libraries). 

Implementation 

• Collected  by:  Local 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly, 

quarterly) 

• Procedures: 

1 . Designate  a staff  member  to  coordinate  the  collection  of  this  statistic. 

2.  Direct  the  designated  staff  member  to  contact  library  staff  members  who  are  in  charge  of 
digital  collection  projects.  Ask  all  library  staff  members  involved  in  any  digital  collection 
projects  as  part  of  their  official  responsibilities  to  fill  out  the  worksheet  in  Appendix  C,  Figure 
C.8,  for  the  reporting  period.  Ask  them  to  estimate  how  much  of  their  time  was  spent  on 
planning,  implementing,  and  managing  digital  collection  projects.  This  information  will  be 
entered  in  the  worksheet  as  FTE  (full-time  equivalent).  A further  breakdown  of  activities 
may  be  necessary  if  the  library  wants  to  have  more  detailed  information  on  the  distribution  of 
efforts. 

Note  that  annual  salary  should  not  be  asked  of  the  staff  members  filling  out  the  worksheet  and 
should  not  include  fringe  benefits.  When  all  the  worksheets  are  collected,  the  salary 
information  will  be  obtained  from  the  library  accounting  or  personnel  department.  Direct 
staff  cost  will  then  be  calculated. 

The  personnel  cost  should  also  include  wages  paid  to  non-salaried  staff,  including  student  and 
other  hourly  workers. 

3.  Cost  of  equipment  should  be  amortized.  For  example,  if  a $3,000  scanner  was  purchased  at 
the  beginning  of  the  reporting  year  and  has  a depreciation  period  of  three  years,  register 
$1,000  as  the  equipment  cost.  Costs  of  software  should  be  reflected  in  full  amounts  based  on 
the  time  of  the  purchase. 

4.  If  a subcontracting  period  is  different  from  the  reporting  period,  prorate  the  amount  for  the 
reporting  period.  If  the  payment  is  based  on  percent  to  completion,  include  only  the  amount 
that  belongs  to  the  reporting  period. 

5.  Use  the  sample  worksheet  in  Appendix  C,  Figure  C.9,  to  calculate  the  total  cost. 

Related  issues:  This  statistic  needs  to  be  collected  and  used  locally  instead  of  across  ARL  member 
libraries  because  of  the  wide  variations  of  the  types  and  features  of  digital  collections  constructed  at  ARL 
institutions. 


4.3.5.  Analysis  of  Suggested  Performance  Measures 

The  overall  rationale  for  the  performance  measures  in  this  study  is  to  provide  a means  for  measuring 
the  proportion  of  services  delivered  through  traditional  channels  relative  to  analogous  services  delivered 
through  electronic  channels.  These  measures  will  help  document  trends  in  service  delivery  for  the 
purpose  of  allocating  staff  and  development  resources  as  well  as  identify  trends  for  strategic  planning  of 
service  delivery. 
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PI  - Percentage  of  Electronic  Reference  Transactions  of  Total  Reference 

Definition:  Percentage  of  annual  electronic  reference  transactions  to  total  reference  transactions.  An 
electronic  reference  transaction  must  include  a question  either  received  electronically  (e.g.,  via  e-mail, 
WWW  form,  etc.)  or  responded  to  electronically.  Count  excludes  phone  and  fax  traffic  unless  either  the 
question  or  answer  transaction  occurs  via  the  described  manner.  It  includes  the  counts  accrued  from 
participation  in  any  local  and  national  projects,  such  as  DigiRef  and  the  Library  of  Congress's  CDRS 
(Collaborative  Digital  Reference  Service). 


Total  reference  - Traditional  reference  counts  (include  face-to-face  reference  transactions, 
telephone  and  fax  reference  counts)  + electronic  reference  transaction  counts 


PI  - Number  of  Electronic  reference  transactions  (TJO  X 100 
Total  reference  transactions 


Rationale:  The  purpose  of  having  a networked  environment  is  to  promote  connectedness.  This 
measure  provides  an  indication  of  a changing  library  environment.  While  in  the  traditional  library 
environment  reference  transactions  were  handled  mainly  through  non-el ectronic  means,  in  the  current 
environment  reference  transactions  can  be  handled  via  various  electronic  means  over  the  Internet.  By 
having  this  as  a measure,  libraries  are  able  to  track  the  development  of  a new  and  emerging  library  service 
and  have  a number  that  fully  represents  reference  activities.  This  measure  may  indicate  how  often  various 
electronic  applications  are  used  in  any  given  period  and  also  assist  decision-makers  in  reallocating 
resources.  Moreover,  this  performance  measure  will  give  administrators  trend  data  on  how  network 
services  are  being  used  and  this  data  can  then  be  used  for  future  planning. 

Implementation 

• Collected  by:  Local 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (monthly, 

quarterly) 

• Procedures: 

1 . If  continuous  collection  of  this  statistic  is  not  possible  or  desirable,  select  a typical  week  (or 
month)  to  run  a sample  study.  It  is  recommended  that  you  sample  a week  in  a different  month 
or  several  months  to  account  for  seasonal  fluctuations.  Extrapolate  based  on  the  sample  data. 

2.  Designate  a staff  member  to  coordinate  the  collection  of  this  measure.  Key  tasks  include 
distributing  a daily  tally  sheet,  collecting  the  daily  tally  sheet,  adding  each  day’s  total  to  a 
weekly  figure,  and  being  available  to  respond  to  data  collection  problems  should  they  arise. 

3.  For  electronic  transactions,  use  the  count  obtained  by  following  the  procedures  for  U 1 . See 
Table  2.7  or  section  3.2.2. 1 of  this  report  for  additional  information  on  Ul. 

4.  Total  the  overall  number  of  transactions. 

5.  Divide  the  number  of  electronic  reference  transactions  by  the  total  number  of  transactions. 

6.  Multiply  by  100. 

7.  Indicate  and  describe  any  additional  methods  used  outside  of  this  definition  and  these 
guidelines. 

Special  considerations:  Count  the  number  of  transactions,  not  the  number  of  questions.  That  is,  if 
one  request  is  emailed  with  three  questions,  it  should  be  counted  as  one  transaction,  not  three. 
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P2  — Percentage  of  Virtual  Library  Visits  of  All  Library  Visits 

Definition:  Number  of  virtual  library  visits  out  of  all  library  visits. 

A virtual  library  visit  is  when  a user  visits  the  library’s  website  or  catalog  for  any  length  of  time  or  for 
any  purpose  from  outside  the  physical  plant  of  the  library,  regardless  of  the  number  of  pages  or  items 
viewed  or  requested.  The  term  “virtual  visit”  excludes  in-library  visits  where  a patron  or  a staff  member 
uses  electronic  resources.  If  a user  looked  at  16  pages  and  54  graphic  images  while  at  a website,  that  user 
registers  one  visit  on  the  web  server.  A visit  is  usually  determined  by  a user’s  IP  address.  Due  to  various 
server  management  issues  and  differing  software,  this  measure  is  an  estimate  of  the  visits  to  the  library 
site. 


All  library  visits  is  the  total  of  the  number  of  virtual  library  visits  plus  the  number  of  physical  visits  to 
the  library  including  branches. 


P2  = # virtual  library  visits  X 100 

Total  library  visits 


Rationale:  People  accessing  the  website  or  catalog  from  outside  the  library  will  reflect  interest  in 
library  services.  The  idea  of  having  network  services  is  to  expand  the  reach  of  libraries  beyond  their 
physical  boundaries,  and  this  performance  measure  can  provide  information  about  how  far  network 
services  are  reaching.  This  figure  will  also  show  the  use  of  the  library  outside  the  regular  place  of 
business,  which  will  be  a more  accurate  depiction  of  library  use.  Having  this  measure  is  important  to 
show  the  continued  relevance  of  library  service  if  physical  attendance  figures  decrease. 

Implementation 

• Collected  by:  Local  and/or  vendors 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (monthly, 

quarterly) 

• Procedures: 

1.  Obtain  the  virtual  library  visits  count  using  the  procedures  for  U5  (virtual  library  visits). 

2.  Obtain  physical  attendance  count  from  turnstile  counts  or  swipe  card  records.  To  the  extent 
possible,  collect  comprehensive  data  from  all  library  branches. 

3.  Combine  the  virtual  visit  count  and  the  physical  attendance  count. 

4.  Divide  the  number  of  virtual  library  visits  by  the  total  library  visits. 

5.  Multiply  by  100.  For  example,  a library  had  1,000  external  virtual  visits  and  9,000  physical 
visits  for  a total  visit  composite  measure  of  10,000.  1,000  virtual  visits  divided  by  10,000 
total  visits  equals  .10  (or  10%). 

P3  — Percentage  of  Electronic  Books  to  All  Monographs 

Definition:  Percentage  of  the  number  of  electronic  books  available  to  users  (through  either  an 
individual  licensing  contract  or  other  consortial  arrangements)  to  all  the  library’s  monographs. 
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Rationale:  Networking  technology  in  libraries  continues  to  improve,  thereby  increasing  user  access  to 
electronic  counterparts  of  some  traditional  sources.  In  the  mid  90s,  networking  and  resource  sharing 
technologies  facilitated  print  and  e-book  access  through  library  networks.  This  performance  measure 
attempts  to  document  the  degree  of  expansion  of  e-books  to  all  monographs.  Libraries  should  use  caution 
while  collecting  this  measure  because  the  definition  of  e-books  is  itself  still  evolving.  This  statistic  is  an 
early  attempt  to  keep  track  of  this  type  of  source  that  is  becoming  increasingly  available. 

Implementation 

• Collected  by:  Local  and  vendors 

• Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (monthly, 

quarterly) 

• Procedures: 

1 . Identify  all  types  of  monographic  materials.  Use  in-house  record-keeping  sources  and  other 
library  sources  to  determine  the  number  of  all  monographs,  including  electronic  books,  non- 
electronic books,  and  other  monographic  materials. 

2.  Identify  electronic  book  types,  including  electronic  books  and  electronic  full-text  aggregate 
services,  using  the  sources  in  step  1 of  the  procedures  for  R3. 

3.  Count  individual  electronic  book  titles.  Record  the  number  of  individual  electronic  books 
from  the  spreadsheet  or  record  the  number  from  another  source. 

4.  Exclude  electronic  reference  books,  i.e.,  publicly  available  electronic  books  that  are  accessed 
for  free. 

5.  Calculate  the  total  number  of  all  monographs,  including  electronic  books,  non-electronic 
books,  and  other  monographic  materials. 

6.  Divide  the  number  of  electronic  books  by  the  number  of  all  library  monographs  (electronic 
and  non-electronic  monographs). 

7.  Multiply  by  100. 

8.  Indicate  and  describe  any  additional  methods  used  outside  of  this  definition  and  guidelines. 


ARL  E-Metrics  Phase  II  Report 


80 


October  2001 


Measures  and  Statistics  for  Research  Library  Networked  Services 


REFERENCES 

Association  of  Research  Libraries  (ARL).  (2001,  May).  ARL  membership  criteria  index.  Available 
at  http://www.arl.org/stats/factor.html. 

Association  of  Research  Libraries  (ARL).  (2001,  October).  ARL  Supplementary  Statistics  1999-2000. 
Washington,  DC:  Association  of  Research  Libraries. 

Bertot,  J.  C.  (2001).  Service  quality  in  the  networked  environment.  Library  Trends , 49(4),  758-775. 

Bertot,  J.  C.,  and  McClure,  C.  R.  (1998).  Measuring  electronic  services  in  public  libraries:  Issues  and 
recommendations.  Public  Libraries,  37(3),  176-180. 

Bertot,  J.  C.,  and  McClure,  C.R.  (1996).  Sailor  network  assessment  final  report:  Findings  and  future 
Sailor  network  development.  Baltimore,  MD:  Division  of  Library  Development  and  Services,  Maryland 
State  Department  of  Education. 

Bertot,  J.  C.,  McClure,  C.  R.,  and  Ryan,  J.  (2001).  Statistics  and  performance  measures  for  public 
library  networked  services.  Chicago,  IL:  American  Library  Association. 

Bertot,  J.  C.,  McClure,  C.  R.,  and  Ryan,  J.  (2000).  Developing  national  library  network  statistics  and 
performance  measures  for  U.S.  public  libraries:  Issues,  findings,  and  recommendations.  Performance 
Measurement  and  Metrics:  The  International  Journal  for  Library  and  Information  Services,  /( 1 ),  15-42. 

Blixrud,  J.  (1998,  April).  Special  issues  in  research  library  measurement.  ARL,  197.  Available  at 
http://www.arl.org/newsltr/197/issues.html. 

Blixrud,  J.  & Jewell,  T.  (1998,  April).  Understanding  electronic  resources  and  library  materials 
expenditures:  An  incomplete  picture.  ARL,  197.  Available  at 
http://www.arl.org/newsltr/197/expend.html. 

Kyrillidou,  M.  (1998,  April).  An  overview  of  performance  measures  in  higher  education  and  libraries. 
ARL,  197.  Available  at  http://www.arl.org/newsltr/197/overview.html. 

Kyrillidou,  M.,  & Crowe,  W.  (1998,  April).  In  search  of  new  measures.  ARL,  197.  Available  at 
http://www.arl.org/newsltr/197/newmeas.html. 

Kyrillidou,  M.  (2000,  December).  Research  library  spending  on  electronic  scholarly  information  is  on 
the  rise.  ARL,  213.  Available  at  http://www.arl.org/newsltr/213/spend.html. 

Lakos,  A.  (1999).  The  missing  ingredient  - culture  of  assessment  in  libraries.  Performance 
Measurement  and  Metrics:  The  International  Journal  for  Library  and  Information  Services,  /(l),  3-7. 

Lindauer,  B.  (1998)  Defining  and  measuring  the  library’s  impact  on  campuswide  outcomes.  College 
& Research  Libraries,  59,  546-70. 

Luther,  J.  (2000).  White  paper  on  electronic  journal  usage  statistics.  Washington,  DC:  Council  on 
Library  and  Information  Resources.  Available  at  http://www.clir.org 


ARL  E-Metrics  Phase  II  Report 


81 


October  2001 


Measures  and  Statistics  for  Research  Library  Networked  Services 


NCLIS  Focus  Group  Meeting.  January,  2001.  Available  at 
http://www.nclis.gOv/statsurv/FrNALRJPT.NCLIS.vendor.FG.l.l  1.01.doc 

Shim,  W.,  McClure,  C.  R.,  & Bertot,  J.  C.  (2000,  December).  Data  gathering  practices  in  the 
networked  environment.  ARL,  213.  Available  at  http://www.arl.org/newsltrZ2 1 3/data.html 

Snowhill,  L.  (2001).  "E-books  and  Their  Future  in  Academic  Libraries.  D-Lib  Magazine  7 (7/8). 
Available  at  http://www.dlib.org/dlib/iulv01/snowhill/Q7snowhill.html 

Sottong,  S.  (2001).  "E-Book  Technology:  Waiting  for  the  'False  Pretender’,"  Information  Technology 
and  Libraries,  20  (2),  72-8.  Available  at  http://www.lita.org/ital/2QQ2  sottong.html. 


ARL  E-Metrics  Phase  II  Report 


82 


October  2001 


227 


Measures  and  Statistics  for  Research  Library  Networked  Services 


APPENDICES 


Appendix  A:  Study  Proposal 

Developing  Statistics  and  Performance  Measures  to  Describe  Electronic 
Information  Services  and  Resources  for  ARL  Libraries: 


Study  Proposal 


Charles  R.  McClure  <cmcclure@lis.fsu.edu>,  Francis  Eppes  Professor,  and 
Director,  Information  Use  Management  and  Policy  Institute 
School  of  Information  Studies 
Florida  State  University 
April  28,  2000 


The  Need  for  Action 


The  rapid  changes  - and  advances  - in  delivering  information  services  and  resources  electronically  pose 
critically  important  challenges  to  the  Association  of  Research  Libraries  (ARL).  These  new  techniques  in  the  delivery 
of  electronic  and  networked  information  have  left  many  ARL  libraries  with  inadequate  data  to  make  a range  of 
decisions  to  provide  cost-effective  and  high  quality  electronic  services  to  their  users.  Basic  questions  such  as  the 
following  have  yet  to  be  adequately  answered: 

• What  methods  can  be  developed  to  determine  who  are  the  users,  what  are  the  uses  of  electronic  information 
• services  and  resources  delivered  by  ARL  libraries,  and  what  is  the  frequency  of  that  use? 

• How  can  ARL  libraries  obtain  timely,  reliable,  comparable,  and  useful  data  from  database  vendors  that 
describe  uses  and  users  of  the  databases?  And,  how  can  ARL  libraries  affect  the  process  that  determines  the 
types  of  data  they  received  from  the  database  vendors? 

• To  what  extent  can  electronic  information  services  and  resources  be  linked  to  a range  of  library  and  higher 
education  outcomes? 

As  demonstrated  at  the  ARL  Scottsdale  meeting  on  New  Measures  in  February  2000,  a number  of  ARL  libraries 
are  working  on  these  and  related  questions.  The  methods  being  developed,  however,  to  answer  these  and  related 
questions  are  not  coordinated  and  are  unlikely  to  be  generalizable  across  the  spectrum  of  ARL  libraries. 

• The  study  prospectus  described  here  provides  a three-phased  approach  to  answer  these  and  other  questions 
regarding  statistics  and  performance  measures  in  the  delivery  of  electronic  services  and  resources.  Upon 
the  successful  completion  of  the  study  ARL  libraries  will  have: 

• A report  that  summarizes  the  current  “state  of  the  art”  of  best  practices  in  ARL  libraries  for  collecting  and 
reporting  statistics  and  performance  measures  related  to  electronic  resources  and  services.  This  report  will 
include  practical  suggestions  and  techniques  that  will  be  of  use  to  ARL  libraries  for  describing  electronic 
services  and  resource  use  and  users. 

• A set  of  statistics  and  performance  measures  that  describe  electronic  information  use,  users,  costs,  and 
staffing.  These  statistics  and  measures  can  assist  ARL  administrators  make  better  decisions  regarding 
deployment,  use,  and  purchase  of  electronic  services. 

• A short,  concise  manual  that  describes  possible  procedures  for  how  to  collect,  analyze,  and  report  data  to 
produce  such  statistics  and  performance  measures.  The  manual  will  stress  practical  and  usable  techniques 
that  can  be  used  in  ARL  libraries  as  well  as  identify  key  issues  to  consider  when  using  such  statistics. 

• A one-day  workshop  for  interested  ARL  staff  offered  by  members  of  the  study  team  that  will  introduce  the 
manual,  describe  how  best  to  use  it,  and  discuss  issues  that  should  be  considered  in  the  use  of  the  manual. 
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• A process  to  collaborate  with  database  vendors  to  develop,  refine,  and  create  statistics  and  performance 
measures  needed  to  describe  database  use,  users,  and  costs.  The  objective  of  this  process  is  to  ensure  that 
database  vendors  understand  the  need  and  importance  of  ARL  libraries  obtaining  reliable  and  comparable 
use  and  user  data  describing  database  use  and  users. 

• Models  that  link  and  describe  relationships  and  possible  impacts  between  electronic  services  and  resources 
and  higher  education  outcomes.  These  models  will  attempt  to  show  where  and  how  electronic  services  and 
resources  contribute  to  accomplishing  selected  higher  education  outcomes. 


These  products  will  be  a first  set  of  tools,  processes,  and  techniques  that  will  assist  ARL  libraries  (I)  better  meet 
the  needs  of  their  users,  and  (2)  make  better  decisions  regarding  the  purchase,  use,  and  deployment  of  electronic 
services  and  resources. 

Leveraging  Existing  Knowledge 

There  is  a limited  number  of  writings  and  still  less  research  available  regarding  the  development  of  statistics  and 
performance  measures  for  the  academic  networked  environment.  This  prospectus  does  not  provide  an  overview  of 
these  writings  - many  of  which  are  listed  on  a bibliography  on  the  New  Measures  webpage  at 
http://www.arl.org/stats/newmeas/e-usage.html.  Generally,  the  literature  identifies  a range  of  issues  and  problems 
regarding  the  development  of  such  statistics.  The  literature  does  not,  however,  adequately  represent  the  practical 
knowledge  and  current  practice  in  this  area  - which  was  identified  at  the  February  28-29,  2000,  ARL  conference  in 
Scottsdale. 

There  is  expertise  outside  the  published  literature  that  can  be  leveraged  by  this  project.  The  author  has  been 
working  to  develop  networked  statistics  and  performance  measures  for  public  libraries  under  a project  funded  by  the 
Institute  for  Museum  and  Library  Services  (IMLS)  http://www.albanv.edu/-imlsstat/.  Currently,  a number  of 
statistics  and  measures  are  being  field  tested  - some  of  which  would  have  clear  applicability  to  academic  libraries. 
Further,  the  project  has  established  important  contacts  and  meetings  with  a number  of  database  vendors  to  discuss 
the  range  and  types  of  data  that  they  might  provide  to  state  and  public  libraries. 

In  addition,  the  author  is  working  with  the  National  Commission  on  Libraries  and  Information  Science  (NCLIS) 
on  the  project,  Testing  National  Public  Library  Electronic  Use  and  Network  Performance  Measures 
http://www.nclis.gov/libraries/lsp/statist.html.  Building  on  the  IMLS  study,  additional  work  is  envisioned  to  meet 
and  collaborate  with  database  vendors  to  determine  the  types  of  data  that  can  be  reasonably  obtained  for  libraries. 
Academic  libraries’  direct  and  formal  involvement  to  participate  in  and  extend  these  efforts  to  address  academic 
library  database  statistical  needs  and  issues  could  build  on  existing  work  and  demonstrate  the  interest  and  concern  of 
the  ARL  community. 


Background 

This  study  prospectus  has  been  under  development  since  fall  1999  when  the  author  discussed  the  possibility  of 
such  a study  with  ARL  staff  and  members  of  the  New  Measures  Initiative 

http://www.arl.org/stats/newmeas/newmeas.html.  Given  an  initial  level  of  interest  with  leaders  in  the  New  Measures 
Initiative  and  ARL  staff,  a preliminary  project  prospectus  was  presented  for  discussion  at  a meeting  held  in 
conjunction  with  the  December  1999  Coalition  for  Networked  Information  (CNI)  meeting  in  Phoenix. 

Based  on  discussions  and  interest  from  the  meeting  at  CNI,  McClure  re-drafted  the  prospectus  and  it  was 
distributed  to  a number  of  ARL  members  and  posted  at  the  ARL  website.  Leaders  of  the  New  Measures  Initiative 
with  ARL  staff  organized  a meeting  on  February  28-29,  2000,  to  discuss  the  appropriateness  of  working  in  the  area 
of  developing  statistics  and  performance  measures  for  networked  information  services  and  resources.  Seventy 
participants  discussed  aspects  of  the  project  at  the  meeting  and  provided  guidance  and  suggestions  for  a study.  This 
Study  Prospectus  is  based  on  the  discussions  and  outcomes  from  that  meeting. 
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A Changing  Context 

Clearly,  the  context  for  delivery  of  library  services  and  resources  is  changing  to  an  electronic  environment. 
While  there  has  been  considerable  discussion  about  the  need  and  importance  of  developing  statistics  and 
performance  measures  to  describe  networked  and  electronic  services  and  resources,  a unified  research  strategy  is 
needed  to  move  forward  on  this  topic. 


The  Key  Term 

The  working  definition  of  network-based  information  services  is:  Those  electronic  information  resources 
and/or  services  that  users  access  on-site  in  the  library,  from  their  office,  dorm,  or  home,  or  from  regional/statewide 
networks.  Examples  of  electronic  networked  resources  include:  local,  regional,  or  statewide  library  hosted  or 
authored  web  sites  or  library-licensed  databases  (e.g.,  Infotrac,  SearchBank,  EbscoHost).  Especially  important  are 
statistics  that  describe  the  use  of  unique  and  often-times  interactive  remote  scientific  and  technical  databases. 

Examples  of  electronic  networked  services  include:  provision  of  access  to  networked  services  such  as  email, 
listservs,  online  reference/assistance,  and  training  in  the  use  of  these  resources  and  services.  In  addition,  libraries 
increasingly  provide  interactive  services  such  as  requesting  services  via  online  forms  (interlibrary  loans,  etc.).  It  is 
likely  that  the  term  “networked  information  resources  and  services”  will  continue  to  evolve  as  the  network  evolves 
and  as  the  study  progresses. 


Growth  in  Electronic  Services  and  Resources 

The  move  to  a networked  environment  has  significantly  increased  the  range  of  services  and  resources  that  the 
library  provides  its  users.  The  library  has  become  a 24  hour  a day  access  point  to  information  services  where  users 
obtain  services  and  resources  on  their  terms  and  when  they  want  such  services  - oftentimes  not  coming  to  the  library 
physically  nor  interacting  directly  with  library  staff.  The  costs  to  provide  these  networked  services  and  resources, 
however,  can  be  significant.  Librarians’  inability  to  develop  reliable  and  accurate  methods  to  describe  these  services 
and  costs  injures  their  ability  to  make  good  resource  allocation  decisions,  meet  user  needs,  and  develop  strategic 
plans  for  the  development  and  operation  of  electronic  services  and  resources. 

On  an  experiential  basis,  most  ARL  librarians  will  describe  the  use  of  their  networked  information  services  with 
terms  such  as  “exponential  growth”  or  “we  can’t  keep  up  with  demand.”  At  the  same  time,  a number  of  ARL 
libraries  have  also  seen  stagnant  or  declining  statistics  of  traditional  indicators  of  library  service  such  is  turn-stil 
counts,  in-house  reference  transactions,  circulation,  etc.  While  there  is  a need  to  develop  new  statistics  such  as 
“virtual  visits,”  “full-text  downloads,”  “electronic  reference  transactions  as  a percentage  of  all  transactions,”  etc., 
there  is  little  agreement  on  how  to  compute  such  statistics  and  measures. 

Example  Possible  Statistics 

The  work  done  by  the  author  and  others  in  the  IMLS  project,  individual  ARL  libraries,  as  well  as  work  under 
way  with  the  NCLIS  study  suggest  that  a number  of  statistics  describing  networked  and  electronic  services  and 
resources  may  be  developed  in  the  following  areas: 

• Count  of  electronic  reference  transactions. 

• Virtual  visits  (sessions)  to  the  library’s  website. 

• Counts  of  high-use  and  low-use  web  pages. 

• Count  of  sessions  on  specific  databases. 

• IP  addresses  for  sessions  on  specific  databases. 

• Time  per  session  on  specific  databases. 

• Tum-aways  per  time  period  per  specific  database. 

• Primary  use  of  selected  electronic  services  and  resources. 

• Hours  of  user  training  on  electronic  services  by  library  staff. 

• Cost  per  session  on  specific  databases. 

• Count  of  full  text  downloads  per  time  period  per  database. 
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• File  size  of  full  text  downloads  per  time  period  per  database. 

• Count  of  on-site  versus  remote  sessions  per  database. 

These  proposed  statistics  are  illustrative  only.  Yet  to  be  accomplished  is  agreeing  on  definitions  and  data  collection 
methods  to  produce  reliable  and  valid  statistics,  determining  the  degree  to  which  such  statistics  can  also  be 
comparable  across  different  libraries,  and  making  linkages  between  such  statistics  and  higher  educational  outcomes. 
In  short,  considerable  work  has  yet  to  be  done. 


Issues  and  Challenges 

Some  of  the  factors  militating  against  the  development  of  networked  statistics  and  performance  measures 
include  the  following: 

• Librarians  do  not  control  access  to  and  use  of  a range  of  data  that  describe  vendor-supplied  information 
services  and  resources.  Some  vendors  are  unwilling  or  unable  to  provide  the  types  of  statistics  and  use  data 
that  librarians  request.  Statistics  and  measures  for  database  use  and  services,  nonetheless,  are  essential. 

• The  rapidly  changing  nature  of  the  networked  environment  also  affects  the  types  of  services  and  resources 
that  can  be  provided  by  libraries.  As  the  networked  services  change,  new  types  of  statistics  and  measures 
may  be  needed. 

• The  level  of  effort  needed  to  collect,  analyze,  and  report  data  to  produce  statistics  and  performance 
measures  for  the  networked  environment  may  be  greater  than  that  needed  to  produce  more  traditional 
statistics. 

• Sometimes  networked  services  costs  and  use  may  be  difficult  to  “unbundle”  if  the  library  obtains  these 
services  through  a consortium.  Costs  can  either  be  hidden  or  be  extremely  difficult  to  allocate  to  individual 
libraries. 

Librarians  may  be  entering  a period  of  time  where  statistics  and  measures  for  networked  services  may  be  useful  for 
two-four  years  and  then  will  have  to  be  re-developed  or  discarded.  Such  an  environment  is  quite  different  than  the 
statistics-collecting  environment  in  which  academic  libraries  previously  existed.  Despite  these  concerns  and  factors, 
ARL  libraries  need  to  move  forward  and  learn  how  best  to  produce  and  use  such  statistics  and  measures  in  this  new 
environment. 

Project  Goals  and  Research  Questions 

One  key  goal  of  this  project  is  to  develop,  test,  and  refine  selected  statistics  and  performance  measures  to 
describe  electronic  services  and  resources  in  ARL  libraries.  A second  goal  is  to  engage  in  a collaborative  effort  with 
selected  database  vendors  to  establish  an  ongoing  means  to  produce  selected  descriptive  statistics  on  database  use, 
users,  and  services.  A third  goal  of  the  project  is  to  develop  a proposal  for  external  funding  to  maintain  the 
development  and  refinement  of  networked  statistics  and  performance  measures.  More  specifically,  the  project  has 
the  following  research  questions: 

• What  existing  techniques  and  approaches  are  being  used  by  ARL  libraries  to  produce  statistics  and 
performance  measures  to  describe  networked  information  services  and  resources?  What  can  be  learned 
from  these  techniques  that  could  be  generalized  to  other  libraries? 

• For  what  purposes  and  for  what  audiences  are  networked  statistics  and  measures  needed? 

• Which  types  of  networked  services  and  resources  should  be  described,  how  should  they  be  defined  and 
operationalized,  and  how  should  the  data  be  collected  to  insure  reliable  and  valid  data? 

• What  performance  or  quality  indicators  are  needed  to  describe  the  impact  and  success  of  such  networked 
services? 
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• How  might  such  statistics  and  measures  be  best  reported? 

• What  linkages  can  be  established  between  these  statistics  and  performance  measures  with  selected  outcomes 
from  higher  education? 

This  list  of  research  questions  suggests  a beginning  perspective  to  direct  project  activities.  These  research  questions 
may  be  revised  as  the  project  proceeds. 

Project  Approach  and  Management 

This  prospectus  assumes  that  a group  of  interested  ARL  libraries  will  work  together  as  a consortium  in 
conjunction  with  a study  team  at  the  Information  Management  Use  and  Policy  Institute  at  Florida  State  University 
http://www.ii.fsu.edu  and  with  ARL  staff.  The  members  of  this  group  would  each  agree  to  make  a commitment  of 
$10,000  to  fund  the  project.  The  study  would  begin  May  12,  2000  and  be  completed  December  2001  (20  months). 
ARL  libraries  would  make  their  contribution  to  ARL;  the  Information  Management  Use  and  Policy  Institute  would 
then  enter  into  a contract  with  the  Association  of  Research  Libraries  to  complete  the  study  as  outlined  in  this 
prospectus. 

Participants  in  the  study  may  have  varying  levels  of  direct  involvement  in  the  project.  Some  may  decide  to  be 
active  participants  in  the  project  in  which  they  would  provide  detailed  descriptions  of  their  current  activities 
regarding  the  collection  and  use  of  networked  statistics;  participate  in  reviewing  and  commenting  on  project 
documents  and  reports;  serving  as  field  sites  to  test  and  refine  statistics  and  performance  measures;  meet  with  other 
participants  to  review  project  activities;  organize  meetings  and  handle  logistics  related  to  those  meetings;  provide 
direct  feedback  and  work  with  the  study  team  to  complete  project  goals;  and  engage  in  specific  data 
collection/analysis  activities  related  to  project  activities.  Other  study  participants  may  decide  not  to  participate  in 
such  activities  or  only  in  those  especially  appropriate  to  their  institution. 

In  each  participating  library  there  would  be  one  person  who  would  serve  as  a liaison  and  as  a single  point  of 
contact  to  coordinate  project  activities.  Depending  on  the  level  of  involvement  by  the  library,  this  person  should  be 
prepared  to  contribute  up  to  25%  of  his/her  time  to  the  project  with  some  additional  assistance  from  others  in  the 
library  from  time  to  time. 

All  libraries  participating  in  the  project  may  appoint  a representative  to  serve  on  the  project’s  advisory 
committee  (AC).  This  group  will  provide  feedback  and  suggestions  to  the  study  team  as  needed;  they  will  be  kept 
informed  of  project  activities;  and  they  will  meet  from  time  to  time  (electronically  and  in-person)  to  discuss  project 
progress  and  issues.  The  advisory  committee  will  be  co-chaired  by  Rush  Miller  and  Sherrie  Schmidt.  Miller  and 
Schmidt  will  serve  as  the  single  point  of  contact  with  the  study  team. 

Three  types  of  staff  time  will  be  committed  to  this  project.  First,  the  study  team  at  Florida  State  University  will 
commit  time  and  effort  to  the  project  as  described  later  in  this  prospectus.  Second,  staff  from  participating  ARL 
libraries  will  be  contributing  time  and  effort  to  the  project  as  described  earlier.  Third,  ARL  staff  will  commit  time 
and  effort  to  the  study  by  assisting  in  selected  data  collection  activities,  insuring  effective  communications  with  ARL 
libraries,  providing  background  information  and  resources,  and  maintaining  a website  to  describe  and  update  project 
activities. 

Project  Phases  and  Activities 

Based  on  discussions  at  the  ARL  New  Measures  invitational  conference  held  in  Scottsdale,  A Z February  28-29, 
2000  the  following  project  phases  and  activities  have  been  developed.  Note  that  two  or  more  activities  within  a 
phase  may  occur  simultaneously.  The  project  activities  related  to  developing  statistics  from  database  vendors  are  an 
ongoing  process  throughout  the  entire  study.  Upon  project  funding  detailed  scheduling  and  tasking  will  occur. 

Phase  I:  Knowledge  Inventory  of  ARL  Libraries  and  organizing  an  ARL  Working  Group  on  Database 
Vendor  Statistics  (May,  2000  - October,  2000) 


ARL  E-Metrics  Phase  II  Report 


87 


October  2001 


Measures  and  Statistics  for  Research  Library  Networked  Services 


The  two  objectives  of  this  phase  are  to  (1)  identify  and  describe  the  current  state  of  the  art  of  statistics  and 
performance  measures  for  networked  services  and  resources  in  ARL  libraries,  and  (2)  organize  an  ARL  Working 
Group  on  Database  Vendor  Statistics  to  begin  discussions  with  database  vendors. 

Activity  1 of  this  phase  will  be  to  conduct  an  inventory  that  will  survey  all  ARL  libraries  as  to  their  current 
practices,  activities,  statistics,  performance  measures,  data  collection  and  reporting  processes.  After  the  survey 
has  been  completed,  the  study  team  will  conduct  site  visits  at  those  libraries  that  appear  to  have  the  most  useful 
information  and  insights  into  study  topics.  The  inventory  will  stress  data  collection,  statistics,  and  performance 
measures  in  the  following  areas: 

• Users  of  networked  information  services  and  resources,  e.g.,  who  are  the  users  of  specific  types  of 
networked  services  and  resources? 

• Uses  of  networked  information  services  and  resources,  e.g.,  what  are  the  applications  and  uses  of  these 
services  and  resources  by  the  users? 

• Staffing  and  training,  e.g.,  how  have  networked  information  services  and  resources  affected  the  staffing  and 
staff  training  in  libraries? 

• Networked  information  services,  e.g.,  electronic  reference  transactions,  electronic  forms  submission,  etc. 

• Cost  analysis  of  networked  information  services  and  resources,  e.g.,  what  are  the  costs  per  transactions  of 
particular  services  such  as  cost  per  full  text  down-load? 

• Vendor-based  database  information,  e.g.,  what  statistics  are  being  compiled  by  which  libraries  from  which 
vendors  with  what  information  and  how  are  those  data  defined?  NOTE  that  this  area  is  an  ongoing  project 
effort  that  continues  throughout  all  phases  of  the  project. 

Initially  these  topics  will  provide  a first  priority  for  inventorying  the  current  knowledge  of  ARL  libraries  related 
to  networked  services  and  resources. 

Due  to  a research  award  received  by  study  team  member  Jeff  Shim,  Florida  State  University,  he  will  be  able 
to  conduct  site  visits  during  the  summer  of  2000  to  selected  ARL  libraries  at  no  cost  to  the  project. 

Activity  2 initiates  a process  whereby  an  ARL  Working  Group  on  Database  Vendor  Statistics  can  begin 
discussions  with  selected  vendors  regarding  a plan  to  identify,  collect,  and  report  various  database  statistics. 
Such  statistics  include,  but  are  not  limited  to,  uses  of  the  databases,  frequency  of  use,  time  of  use,  log-ins  to 
particular  titles,  IP  addresses  of  log-ins,  tum-aways,  etc. 

Tasking  for  this  phase  includes  the  establishment  of  an  ARL  Working  Group  on  Database  Vendor  Statistics: 
having  the  group  define  key  issues  and  objectives;  coordinate  activities  with  ICOLC  and  other  appropriate 
groups;  propose  possible  guidelines  and  procedures  for  vendor-produced  database  statistics;  begin  meeting  with 
selected  database  vendors;  and  develop  a process  to  clarify  needs  and  expectations  by  both  the  vendors  and  the 
ARL  Working  Group. 

Products  from  Phase  I:  The  product  from  Phase  I will  be  an  interim  report  that: 

• Describes  the  knowledge  and  best  practices  currently  in  use  by  ARL  libraries  regarding  statistics  and 
performance  measures  for  electronic  information  services  and  resources; 

• Describes  the  range  and  types  of  data  being  obtained  by  ARL  libraries  from  the  various  database  vendors; 

• Describes  key  issues  that  will  need  to  be  resolved  in  producing  statistics  and  performance  measures;  and 
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• Presents  a status  report  on  the  activities  of  the  ARL  Working  Group  on  Database  Vendor  Statistics,  its 
objectives,  and  its  next  steps  in  Stage  II. 

In  addition,  members  of  the  study  team  will  provide  an  Executive  Briefing  to  project  participants  in  conjunction 
with  the  Fall,  2000  ARL  meeting. 

Phase  II:  Development  of  statistics  and  performance  measures  (November,  2000- June,  2001) 

The  objective  of  this  stage  is  to  develop  tools,  data  collection  processes,  statistics,  and  performance  measures  to 
describe  services  and  resources  in  the  networked  environment. 

Activity  1 will  develop  an  approach  to  take  the  knowledge  learned  from  Phase  I into  a research  methodology  to 
develop,  define,  and  propose  possible  statistics  and  measures.  In  short,  the  activity  will  produce  a detailed 
tasking  and  methodology  by  which  statistics  and  performance  measures  can  be  developed,  tested,  and  refined. 
This  activity  will  also  propose  data  collection  techniques  and  instruments  for  use  by  participating  libraries  to 
produce  statistics  and  measures  in  each  of  the  areas  identified  in  Phase  I.  The  process  will  draw  heavily  upon 
input  and  advice  (in  an  iterative  fashion)  from  participating  libraries. 

Activity  2 will  be  the  field-testing  of  these  proposed  data  collection  techniques,  statistics,  and  measures.  The 
study  team  anticipates  that  four-six  participating  libraries  will  serve  as  field  sites  to  test  the  approaches 
developed  in  phase  I.  A process  will  be  developed  for  each  library  to  test  some  portion  of  the  proposed  statistics 
and  report  on  the  efficacy  of  the  process  as  well  as  the  statistics  and  measures  themselves. 

Activity  3 will  be  the  analysis  of  the  field  test  results  and  the  writing  of  a short  manual  that  describes  the  process 
for  data  collection,  the  statistics,  and  the  performance  measures.  The  study  team  will  determine  what  appears  to 
be  working  well  in  terms  of  process  and  what  needs  additional  work  and  revision.  Depending  on  the  results 
from  the  field  test,  some  additional  work  and  refinement  on  selected  statistics  and  measures  may  be  needed  prior 
to  writing  the  data  collection,  statistics,  and  measurement  manual. 

Activity  4 of  this  phase  is  to  develop  a model  that  integrates  the  statistics  and  measures  into  (1)  higher  education 
educational  outcomes,  research  outcomes,  and  service  outcomes,  and  (2)  library  educational  outcomes,  research 
outcomes,  and  service  outcomes.  In  fact,  this  phase  will  be  under  development  throughout  the  entire  phase  as 
development  of  statistics  and  measures  must  occur  in  the  broader  context  of  their  purposes  and  their  relationship 
with  higher  education  outcomes.  Findings  from  Phase  I will  inform  the  development  of  such  models. 

Activity  5 of  this  Stage  is  to  continue  activities  of  the  ARL  Working  Group  on  Database  Vendor  Statistics.  This 
effort  is  ongoing  throughout  Phase  II.  During  this  Phase  we  would  expect  that  meetings  between  the  ARL 
working  group  and  the  database  vendors  would  be  moving  toward  agreement  on  data  element  definitions  and 
terms,  to  specific  statistics  and  data  that  can  be  collected,  and  methods  for  reporting  these  data  to  libraries.  The 
study  team  would  expect  to  include  these  data  collection  techniques  in  the  field  test  described  in  activity  3 
above. 

Products  from  Phase  II  include; 

• A written  methodology  to  develop  and  field  test  data  collection  techniques,  statistics,  and  performance 
measures; 

• A short  concise  written  manual  that  describes  how  these  statistics  and  measures  can  be  produced,  that  will 
include: 

introduction  to  the  importance  and  need  for  statistics  and  performance  measures  that 
describe  electronic  services  and  resources; 

data  collection  techniques  and  methods  for  each  of  the  statistics  and  performance  measures; 
issues  to  be  considered  in  using  and  interpreting  these  statistics  and  performance  measures; 
recommendations  for  future  work  in  the  development  of  new  or  refinement  of  existing  statistics  and 
performance  measures;  and 


ARL  E-Metrics  Phase  II  Report 


89 


October  2001 


Measures  and  Statistics  for  Research  Library  Networked  Services 


appendices  of  data  collection  instruments  developed  and  tested  during  the  field  tests; 

• A written  status  report  on  the  activities  and  accomplishments  of  the  ARL  Working  Group  on  Database 
Vendor  Statistics,  and  issues/next  steps  to  be  addressed;  and 

• A one-day  workshop,  held  in  conjunction  with  ALA  or  an  ARL  meeting  to  present  the  manual  and  discuss 
project  findings  during  the  late  spring  of  2001. 


Phase  III:  Institutionalizing  Statistics  and  Performance  Measures  (July  2001-December  2001) 

The  objective  of  this  phase  is  to  develop  mechanisms  and  processes  that  insure  the  ongoing  development  of 
networked  statistics  and  performance  measures.  This  objective  includes  building  and  promoting  infrastructure 
in  ARL  and  ARL  libraries  to  continue  the  development  and  use  of  such  statistics  and  measures. 

Activity  1 of  this  phase  is  to  develop  a research  proposal  to  obtain  external  funding  to  continue  research  and 
field-testing  related  to  networked  information  statistics  and  measures.  The  study  team  will  work  with  ARL  staff 
and  the  advisory  committee  in  the  development  of  this  proposal  and  the  identification  of  appropriate  funding 
bodies  that  may  be  interested  in  supporting  continued  work  in  this  area.  Specific  research  questions  and 
initiatives  in  this  proposal  will  result  from  findings  and  activities  in  Phase  II. 

Activity  2 of  this  phase  is  to  develop  a number  of  training  modules  and  training  support  systems  (both  in  print 
and  electronically)  that  ARL  and  ARL  libraries  can  use  to  assist  staff  understand  the  importance  of  the  new 
statistics  and  measures  developed  as  part  of  this  project,  as  well  as  help  them  on  a very  practical  level  collect, 
analyze,  and  report  quality  data.  The  study  team  anticipates  developing  and  testing  these  modules  but  assumes 
that  ARL  members  or  ARL  staff  would  be  actively  engaged  in  the  instruction. 

Activity  3 of  this  phase  is  to  continue  refining  and  testing  the  model(s)  that  integrate  the  statistics  and  measures 
into  (1)  higher  education  educational  outcomes,  research  outcomes,  and  service  outcomes,  and  (2)  library 
educational  outcomes,  research  outcomes,  and  service  outcomes.  The  study  team  anticipates  developing  a 
process  to  validate  these  models  and  determine  the  potential  usefulness  of  the  model(s)  to  describe  impacts  from 
networked  information  services  and  resources. 

Activity  4 of  this  phase  will  continue  activities  of  the  ARL  Working  Group  on  Database  Vendor  Statistics.  This 
effort  is  ongoing  throughout  Phase  III.  During  this  Phase  we  would  expect  that  meetings  between  the  ARL 
Working  Group  and  the  database  vendors  would  have  reached  agreement  on  data  element  definitions  and  terms, 
on  specific  statistics  and  data  that  can  be  collected,  and  methods  for  reporting  these  data  to  libraries.  The  study 
team  would  expect  to  assist  the  ARL  Working  Group  extend  the  number  of  database  vendors  agreeing  to  the 
guidelines  that  would  have  resulted  for  data  collection  and  statistics. 

Products  from  Phase  III  include: 

• A proposal  that  can  be  submitted  to  potential  fundors  to  continue  research  and  development  on  statistics  and 
performance  measures  in  the  networked  environment  and  their  potential  impacts  on  a range  of  higher 
education  outcomes; 

• A revised  and  updated  description  of  database  statistics  and  performance  measures.  Based  on  the  additional 
activities  of  the  working  group  and  discussions  with  database  vendors  we  would  expect  to  be  able  to  build 
upon  the  database  statistics  developed  in  phase  II  and  expand  and  refine  them; 

• Instructional  modules  that  can  assist  ARL  and  ARL  member  institutions  train  staff  as  to  the  importance  and 
process  for  collecting  and  analyzing  networked  statistics  and  performance  measures;  and 

• A final  report  of  guidelines  and  issues  yet  to  be  resolved  as  agreed  to  by  ARL  and  participating  database 
vendors  on  the  collection  and  reporting  of  selected  statistics  and  data  elements  from  these  databases. 
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It  should  be  recognized  that  some  fine-tuning  of  these  phases  and  activities  may  occur  as  the  project  proceeds.  Such 
fine-tuning  will  be  done  with  the  advice  of  the  advisory  committee. 

Study  Team  Qualifications 

Charles  R.  McClure  will  serve  as  the  Principal  Investigator  for  the  study.  Detail  on  his  background,  experience, 
and  examples  of  recent  projects  he  has  completed  can  be  found  at  <http://slis-two.lis.fsu.edu/~cmcclure>. 

Information  about  the  Information  Use  Management  and  Policy  Institute  for  which  he  serves  as  director  can  be 
obtained  at  http://www.ii.fsu.edu.  He  has  a proven  track  record  of  managing  projects  successfully  and  has  worked 
with  public  libraries  in  a range  of  areas  including  financial  assessment,  planning  and  evaluation  of  information 
services,  information  technology  management,  and  resource  sharing.  Currently,  with  John  Carlo  Bertot,  he  is 
completing  a study  funded  by  the  Institute  for  Museum  and  Library  Services  (IMLS)  that  will  produce  national 
statistics  and  performance  measures  to  describe  public  library  services  and  resources  in  the  networked  environment 
<http://www.albanv.edu/~imlsstat/>. 

John  Carlo  Bertot  is  an  Associate  Professor  in  the  School  of  Information  Science  and  Policy  at  SUNY  Albany. 
He  has  worked  with  and  Charles  R.  McClure  successfully  on  a number  of  library  studies,  including  the  Public 
Libraries  and  the  Internet  national  surveys  from  1994  through  1998,  and  Internet  project  impact  studies  in 
Pennsylvania  {Evaluation  of  the  Online  at  PA  Libraries  Project:  Public  Access  to  the  Internet  through  Public 
Libraries)  and  California  {The  Importance  of  California  Public  Libraries  in  Increasing  Public  Access  to  the 
Internet ).  Bertot  and  McClure  have  also  collaborated  on  a number  of  technology  planning  and  evaluation  projects. 
Additional  background  information  on  Bertot’s  experience  and  skills  can  be  found  at 
<http://www.albanv.edu/~icbertot/>. 

Jeff  Shim,  Assistant  Professor  of  Information  Studies  at  Florida  State  University  and  a Research  Associate  at  the 
Information  Use  Management  and  Policy  Institute  will  also  serve  on  the  study  team.  Shim  recently  completed  his 
Ph.D.  from  Rutgers  University  where  his  primary  research  centered  on  an  analysis  of  ARL  statistics.  He  teaches  in  a 
range  of  areas  related  to  the  management  of  information  technology.  He  is  especially  knowledgeable  about 
academic  library  statistics,  statistical  techniques  for  describing  services  and  resources,  and  understands  information 
technology- based  services  provision. 

Additionally,  we  expect  that  a number  of  Ph.D.  students  and  other  graduate  students  from  the  School  of 
Information  Studies  at  Florida  State  University  will  be  working  on  the  project  as  research  assistants.  These 
additional  staff  will  be  involved  on  the  project,  oftentimes  with  no  additional  cost  to  the  project. 

Budget  and  Financial  Arrangements 

The  budget  for  this  project  is  $199,990  and  is  summarized  in  Figure  1.  Every  effort  has  been  made  to  keep 
overall  costs  to  a minimum  and  use  budgeted  monies  as  effectively  as  possible. 

Figure  1.  Budget  May,  2000  - December  2001 

Personnel*  $143,844 

1.  Charles  R.  McClure,  Principal  Investigator 

2.  Jeff  Shim,  Associate  Director 

3.  John  Bertot,  Associate  Director 

4.  Bruce  Fraser,  Project  Manager 

5.  Research  Assistants 

* Summer  (May-August)  will  have  greater  average 
study  team  time  commitment  than  the  academic  year 

Benefits  on  Personnel  28,446 

Study  Team  Travel  to  ARL  Libraries  and  Meetings  19,000 
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[16  trips  @$1200  per  trip] 

National  Workshop  [at  end  of  Phase  II] 

1,250 

Communications  [supplies,  telephone,  copying,  etc.] 

7,250 

Total 

$199,990 

The  primary  expenses  will  be  for  study  team  personnel,  travel  expenses  to  conduct  field  tests  and  participate  in 
various  meetings,  and  various  communications  support.  The  study  would  be  conducted  as  a fixed-price  contract. 
One-fourth  of  the  project  costs  will  be  paid  upon  formal  agreement  to  conduct  the  study,  with  one-fourth  of  the 
project  costs  paid  at  the  conclusion  of  each  of  the  three  stages. 

Project  Communications 

The  study  team  anticipates  regular  and  ongoing  communication  with  participating  institutions  as  well  as  all  ARL 
libraries  via  the  ARL  New  Measures  Project  Initiative  Website  (on  the  ARL  or  another  Website).  ARL  staff  as  well 
as  possibly  other  staff  from  ARL  participating  libraries  will  have  responsibility  to  mount  project  information, 
updates,  project  reports,  issue  papers,  and  other  items  on  that  website. 

In  addition,  a discussion  list  related  to  the  project  will  be  operated  and  maintained  by  ARL  for  regular  posting  of 
information  and  for  the  exchange  of  ideas  and  views  related  to  the  project  and  the  development  of  statistics  and 
performance  measures  related  to  the  networked  environment.  Regular  meetings  (either  in  conjunction  with  other 
professional  meetings  or  electronically)  will  occur  with  the  advisory  committee  to  discuss  project  activities.  The 
project  Advisory  Committee  Chairs  and  project  liaisons  (Miller  and  Schmidt)  will  keep  other  committees  at  ARL 
apprised  of  project  activities  as  needed. 


Use  of  Project  Information  and  Findings 

The  study  team  reserves  the  right  to  use  data  and  findings  from  this  project  in  other  future  studies  and  research 
efforts.  Indeed,  much  of  the  insight  and  information  that  the  study  team  brings  to  this  effort  is  a result  of  years  of 
previous  research  and  related  projects.  In  addition,  McClure,  Bertot,  and  Shim  reserve  the  right  to  publish  papers 
(hopefully  in  conjunction  with  project  participants)  and  otherwise  use  information  from  this  project  for  other 
educational  and  instructional  purposes.  Any  papers  submitted  for  publication  would  be  provided  to  the  liaison  for 
review  and  comment  prior  to  actual  publication  and  would  acknowledge  the  support  of  the  project.  Data  and  related 
findings  from  the  project  would  be  reported  in  aggregate  form  only  and  would  not  be  linked  to  individual  libraries. 

Importance  of  the  Project 

To  some  extent,  ARL  libraries  are  under-representing  and  under-counting  the  range  of  services  and  resource 
provision  in  which  they  are  engaged  because  there  are  no  agreed  upon  means  to  count  and  describe  service  provision 
in  the  networked  environment.  In  addition,  resource  allocation  decisions  are  further  complicated  by  a limited 
understanding  of  the  use  and  costs  of  services  provision  in  the  networked  environment.  Given  this  situation,  library 
administrators  are  oftentimes  unable  to  demonstrate  the  importance  and  impact  of  such  networked-based  services. 
Such  is  true  for  individual  libraries  as  well  as  for  ARL  libraries  as  a group. 

Clearly,  the  problems  and  issues  identified  in  this  prospectus  regarding  counts  and  measures  of  services  and 
resources  in  the  networked  environment  will  not  disappear  in  the  near  future.  Indeed,  these  issues  are  only  likely  to 
increase  in  importance  as  the  networked  environment  evolves  with  a range  of  new  services  and  resources.  Support 
for  this  project  provides  ARL  and  a group  of  ARL  libraries  to  take  a leadership  role  in  the  process  of  developing 
statistics  and  measures  for  services  and  use  in  the  networked  environment.  It  provides  a basis  for  ARL  libraries  to 
formally  begin  work  with  selected  database  vendors  to  reach  agreement  on  possible  data  reporting  activities. 
And... the  project  will  begin  to  develop  linkages  between  these  statistics  and  performance  measures  and  a range  of 
higher  education  outcomes. 
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Most  ARL  libraries  need  such  statistics  and  measures  now.  The  sooner  work  is  initiated  on  a project  such  as  that 
outlined  in  this  prospectus,  the  sooner  such  statistics  and  measures  can  be  used  to  support  resource  allocation 
decisions,  services  provision  and  assessment,  and  strategic  planning. 
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Appendix  B.  List  of  Statistics  and  Measures  Considered 


Table  B.l  List  of  Statistics  and  Measures  Considered 


Categories 

E-Metrics  vl.O 
(Initial  List) 

E-Metrics  v.  1 . 1 
(Revised  List) 

E-Metrics  v.2.0 
(Field  Test  List) 

Resources 

• Number  of  electronic  full-text 
journals  (hosted  by  library) 

• Number  of  electronic  full-text 
journals  (through  subscription) 

• Number  of  librarians 
providing  electronic  reference 

• Number  of  public  access 
workstations 

• Number  of  electronic  full-text 
periodicals  (hosted  by  library) 

• Number  of  electronic  full-text 
periodicals  (through 
institutional  subscription) 

• Number  of  electronic  full-text 
periodicals  (through  consortia 
and  other  arrangements) 

• Number  of  electronic 
reference  databases  (through 
institutional  subscription) 

• Number  of  electronic 
reference  databases  (through 
consortia  and  other 
arrangements) 

• Number  of  electronic  books 

• Number  of  staff  providing 
electronic  reference 

• Number  of  public  access 
workstations 

• Number  of  electronic  full-text 
journals  (institutional) 

• Number  of  electronic  full-text 
journals  (consortia) 

• Number  of  electronic 
reference  sources  (institutional) 

• Number  of  electronic 
reference  sources  (consortia) 

• Number  of  electronic  books 
(institutional) 

• Number  of  electronic  books 
(consortia) 

Use 

• Logins  (sessions) 

• Queries  (searches) 

• Tum-aways  (requests  exceed 
simultaneous  user  limit) 

• Items  examined  (viewed, 
downloaded,  emailed,  printed) 

• Total  user  connection  time  to 
vendor  databases 

• Virtual  visits  to  networked 
library  resources 

• Electronic  reference 
transactions 

• Number  of  people  participated 
in  user  instruction  on  electronic 
resources 

• Number  of  logins  (sessions)  to 
networked  library  resources 

• Electronic  reference 
transactions 

• Number  of  Logins  (sessions) 
to  electronic  databases 

• Queries  (searches) 

• Total  connection  time  to 
electronic  databases 

• Items  examined  (viewed, 
downloaded,  emailed,  printed) 
to  electronic  databases 

• Tum-aways  (requests  exceed 
simultaneous  user  limit) 

• Number  of  people  participated 
in  user  instruction  on  electronic 
resources  and  services 

• Number  of  electronic 
reference  transactions 

• Number  of  logins  (sessions)  to 
electronic  databases 

• Number  of  queries  (searches) 
in  electronic  databases 

• Items  examined  in  electronic 
databases 

Cost 

• Cost  of  electronic  database 
subscriptions 

• Cost  per  items  examined 
(subscribed  databases) 

• Cost  of  electronic  files  (one- 
time/monographic purchase) 

• Cost  of  electronic  full-text 
periodicals  subscriptions 

• Cost  of  electronic  reference 
databases  subscription 

• Library  contribution  to 
consortia  for  electronic 
databases 

• Cost  of  electronic  full-text 
journals 

• Cost  of  electronic  reference 
sources 

• Cost  of  electronic  books 

• Library  expenditures  for  bib. 
utilities,  networks,  and 
consortia 

• External  expenditures  for  bib. 
utilities,  networks,  and 
consortia 

Local  Digital 
Collection 

• Cost  of  internal  digital 
collection  construction 

• Cost  of  internal  digital 
collection  construction 

• Size  of  library  digital 
collection 

• Use  of  library  digital 
collection 

• Cost  of  digital  collection 
construction  and  management 
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Categories 

E-Metrics  vl.O 
(Initial  List) 

E-Metrics  v.  1 . 1 
(Revised  List) 

E-Metrics  v.2.0 
(Field  Test  List) 

Performance 

Measures 

• Percentage  of  electronic 
reference  transactions  of  total 
reference 

• Percentage  of  electronic 
materials  use  of  total  library 
materials  use 

• Percentage  of  remote  library 
visits  of  all  library  visits 

• Ratio  of  public  access 
workstations  to  university 
population  (number  of  faculty, 
staff,  and  students) 

• Percentage  of  of  electronic 
reference  transactions  of  total 
reference 

• Percentage  of  of  electronic 
materials  use  of  total  library 
materials  use 

• Percentage  of  of  remote 
library  visits  of  all  library  visits 

• Percentage  of  of  electronic 
titles  to  all  periodicals 

• Percentage  of  of  electronic 
books  to  all  monographs 

• Ratio  of  public  access 
workstations  to  university 
population 

• Cost  per  items  examined  in 
individually  subscribed 
databases 

• Percentage  of  of  electronic 
reference  transactions  of  total 
reference 

■ Percentage  of  of  electronic 
materials  use  of  total  library 
materials  use 

■ Percentage  of  of  remote 
library  visits  of  all  library  visits 

• Percentage  of  of  electronic 
books  to  all  monographs 
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Appendix  C.  Forms 

Figure  C.l  (for  C1-C3) 

SAMPLE  ELECTRONIC  RESOURCE  COST  REPORT  FORM 

Reporting  Period: 


Name  of  library: 


Resource/  Consortium  Name  and  Type 
: WA,  Fulkext  journals  (1) 

Reference  sources  (2) 
Electronic  books  (3) 

Cost 

Comments 

Sub  Total  (1) 

Sub  Total  (2) 

Sub  Total  (3) 

Grand  Total  (1+2+3) 
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Figure  C.2  (for  C4) 

SAMPLE  CONSORTIA  EXPENDITURE  REPORT  FORM 


Reporting  Period: 
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Figure  C.3  (for  C5) 

SAMPLE  CONSORTIA  FUNDING  REPORT  FORM 


Reporting  Period: 


Name  of  library: 


Consortium  Name 

Total  Funding 
Amount 

Amount  Attributable  to 
the  Library 

Comments 

Figure  C.4  (for  Dl) 

SAMPLE  LIBRARY  DIGITAL  COLLECTION  INVENTORY  REPORT  FORM 

Reporting  period: 

Name  of  library: 


Visual 

h Materials  ’ 

. Texts  ,'ij 

Audio/Video/Multimedia 

Total  - . • 

Titles 

(i) 

Items 

(2) 

Titles 

(3) 

Titles 

(4) 

Titles 

(1+2+3+4) 

Size  (GB) 
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Figure  C.5  (for  Dl) 


SAMPLE  DIGITAL  COLLECTION  ITEMS  ADDED  REPORT  FORM 


Reporting  period: 
Name  of  library:  _ 


Figure  C.6  (for  Dl) 

SAMPLE  LIBRARY  DIGITAL  COLLECTION  REPORT  FORM 

Reporting  Period: 


Name  of  library: 


Project 
Name  tMl 

Information  Type 

1. ETD  : : | - H 

2.  Visual  Materials 

3.  Texts  ..i  >■> -4^. 

4.  Audlo/Video/Multimedia 

Server 

Nanie 

Directory  Location 

.I  rr  *.  ^ ; 

No.  of 
Titles 

• Size  *: 
(GB). 
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Figure  C.7  (for  D2) 

SAMPLE  DIGITAL  COLLECTION  ACCESS  REPORT  FORM 

Reporting  Period 


Name  of  library: 


Project  ’> 
Name 

Server 
iL.Nanje  __ 

Directory 

Location 

Title  Access 
Count 

Item  Access 
Count 

Total 

Searches 

Comments 

Figure  C.8  (for  D3) 

SAMPLE  DIGITAL  COLLECTION  COST  REPORT  FORM  -Personnel 


Reporting  Period 
Name  of  library: 


Name  iWi 

Position  1 

{Annual  Salary-} 

ft§* 

{Staff  Cost} 
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Figure  C.9  (for  D3) 


SAMPLE  DIGITAL  COLLECTION  COST  REPORT  FORM 


Reporting  Period 
Name  of  library: 


Project  Name 

Expense  Type 

Amount 

Project  (name)  Total 

, 7 - - ■ ' ' * " .-JS  - ’ "-r 

Project  (name)  Total 

• *' •./ : •.  - v 

Library  Total 
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Information  Use  Management  and  Policy  Institute 


gw? 


November  20,  2001 


Dear  E-Metrics  Project  Participants: 

The  attached  tables  summarize  information  regarding  usage  statistics  of  electronic  content 
services  provided  to  the  E-Metrics  project  team  as  part  of  the  vendor  statistics  field  testing  during  May 
and  June  2001 . The  information  was  provided  voluntarily  by  the  database  vendors  as  listed  in  the  tables 
to  aid  understanding  of  usage  statistics  being  provided  by  these  vendors. 

The  tables  should  provide  a context  in  which  future  discussions  to  improve  usage  statistics  can 
take  place.  Please  note  that  usage  statistics  from  vendors  change  constantly  and  information  herein  may 
not  reflect  current  practices. 

Please  contact  Jeff  Shim  (wshim@lis.fsu.edu)  if  you  have  any  questions  regarding  the  contents  of 
the  tables. 


Wonsik  “Jeff’  Shim 
Assistant  Professor 

Information  Use  Management  and  Policy  Institute 
Florida  State  University 


School  of  Information  Studies 
Florida  State  University 
Information  Institute 
http://www.ii.fsu.edu 
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Table  1.  Searches 


Vendor 

Category 

Information 

Academic 

Press 

(IDEAL) 

Definition 

“Not  Provided” 

Additional 

Information 

“Not  Provided” 

Bell  & 
Howell 

Definition 

Search  count  is  incremented  every  time  the  search  button  is  clicked. 

Additional 

Information 

■ The  information  provided  is  the  total  number  of  searches  in  each  search  mode 
broken  out  by  database,  location,  and  userid. 

- Subsequently  clicking  next  to  retrieve  the  next  set  of  results  does  not 
increment  the  search  count 

- Subsequently  browsing  of  issues  of  a journal  and/or  documents  within  the 
journal  does  not  increment  the  search  count 

- Searching  and  navigating  within  the  topic  tree  does  not  increment  the  search 
count. 

EBSCO 

Definition 

Search  can  be  counted: 

- for  each  search  of  an  EBSCO  host  database 

- once  for  each  database  when  a single  search  is  applied  to  multiple  databases 

- for  each  persistent  link  followed  (a  search  by  accession  number  is  performed) 

■ for  each  database  search  conducted  by  browsing  authority  files  such  as 

subjects,  journals,  authors,  etc 

- when  a subject  link/an  author/a  table  of  contents  are  clicked  on  the  full  display 

of  an  article.  (The  link  is  the  volume/issue  enumeration) 

Additional 

Information 

■ The  total  search  count  accumulated  for  customers 

- Searches  against  multiple  databases  will  increment  for  each  database 

■ Searches  repeated  will  count  each  time  they  are  performed 

Elsevier 

(Science 

Direct) 

Definition 

■ Searches  can  be  performed  using  the  basic  and  advanced  modes.  [Search 
activity]  report  shows  the  usage  of  each  of  these  two  search  modes  and  the  total 
number  of  searches  submitted. 

Additional 

Information 

■ At  ScienceDirect  a searchform  with  operators  and  terms  used  is  not  sent  to  the 
server  but  to  special  high  speed  databases.  Databases  subsequently  send  search 
results  to  the  server. 

■ Searches  can  be  counted  successful  and  intentional  end-user  requests  after 
filtering. 

Gale  Group 

Definition 

A search  is  counted  for  each  search  form  submitted  or  search  link  selected  in 
any  database 
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Vendor 

Category 

Information 

- 

Additional 

Information 

■ An  InfoMark  search  is  counted  each  time  it  is  executed. 

■ If  the  search  engine  needs  to  perform  additional  searches  or  re-execute  a 
search,  then  additional  searches  are  counted  (such  as  when  a user  reuses  a 
search  result  from  an  expired  session). 

■ Search  counts  are  higher  on  our  InfoTrac  Web  usage  reports  due  to  the  fact 
that  we  “replay”  searches.”  Replays”  are  re-executed  searches,  (and  search 
counts  increase  accordingly).  Cases  when  searches  are  replayed  are: 

1)  A user  switches  databases  and  goes  back  to  a search  in  a previous  database 
For  example,  if  a user  goes  through  these  steps: 

1 . User  is  in  Database  A and  performs  a search. 

2.  User  opens  Database  B (A  is  closed,  B is  opened). 

3.  User  goes  back  and  clicks  on  a Database  A search  (B  is  closed,  A is 
opened,  search  is  replayed.) 

So,  A is  re-opened  and  enough  done  to  get  the  user  to  the  point  desired 
(replaying  the  search  to  get  the  citation  list).  Not  everything  that  the  user 
previously  performed  is  replayed.  (Nested  searches  are  replayed  in 
sequence.  Ex:  search,  read  article,  click  on  doc  link  search.) 

2)  Upon  restarts  of  after  a session  time-out. 

After  6 minutes  of  user  inactivity,  the  user  interface  (UI)  calls  the  back-end 
server  (ITS)  to  close  down  the  associated  session.  If  later,  the  user 
continues  then  a new  backend  ITS  session  is  started.  If  the  user  clicks  on  a 
previous  search,  then  a search  is  executed  (in  essence  replayed  and 
recounted). 

A UI  session  can  be  timed-out  and  if  restarted,  can  replay  and  recount  a 
search.  The  time-out  time  has  been  chosen  in  consideration  of 
simultaneous  users  wanting  to  get  on.  Another  case  where  it  replays  and 
recounts  a search. 

1.  User  performs  a search  and  gets  back  a citation  list. 

2.  User  looks  at  an  article  from  that  list. 

3.  User  clicks  on  a document  link  which  returns  a citation  list. 

4.  User  goes  back  and  clicks  on  the  first  citation  list,  in  A.  (This  action 
will  replay  the  search  to  get  the  citation  list.) 

Lexis-Nexis 

Definition 

■ Searches  submitted  refers  to  the  number  of  times  the  specific  search  form 
URL  was  accessed  and  the  search  button  was  clicked  on. 

Additional 

Information 

■ Assume  that  access  of  specific  URL  equals  document  retrieval/search. 

netLibrary 

Definition 

A patron  performed  a search  across  the  eBook  collection  by  filling  in  a web 
form  and  clicking  a search  button  on  either  the  Quick  Search  field  present  on 
most  netLibrary  web  pages,  the  Power  Search  form  or  the  Command  Search 
form. 

Additional 

Information 

■ This  statistics  reports  the  number  of  searches  performed. 

■ All  searches  are  reported:  keyword  searches  and  full  text  searches. 

■ Statistics  are  reported  as  a summary  and  by  time  of  day,  day  of  week. 
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Vendor 

Category 

Information 

OCLC 

(FirstSearch) 

Definition 

FirstSearch  counts  a search  for  each  of  the  activities  listed  below. 

• A search  in  multiple  databases  is  counted  in  each  database. 

Ex)  if  3 databases  are  included  in  a search,  3 searches  are  counted. 

• Searching  for  a word  or  phrase  using  a search  screen,  previous  searches 
screen,  or  other  screen  on  which  searches  can  be  entered. 

■ Redoing  a search  with  added  words,  phrases,  or  search  limits. 

• Clicking  a link  that  returns  search  results. 

Ex)  clicking  a subject  or  author  link  in  a record  or  Expand  Search  screen, 
clicking  a link  in  a Limit  Results  screen,  or  clicking  a subject  heading  link  in 
a Preferred  Subject  Headings  screen. 

• Clicking  the  results  link  for  a previous  search  or  combining  two  or  more 
previous  searches. 

■ Viewing  or  e-mailing  a full-text  document  when  using  the  per-search  method 
to  purchase  full  text.  Each  document  viewed  or  e-mailed  counts  as  five 
searches.  If  you  view  and  e-mail  the  same  document,  it  is  counted  once  as  five 
searches. 

Ex)  The  following  are  WorldCat  database  search  examples. 

• First  example  (3  to  5 searches  counted): 

- Search  for  fetal  alcohol  syndrome  (one  search). 

- Redo  the  search  adding  the  term,  and  education  (a  second  search). 

- Redo  the  search  using  limits  (1980  to  the  present,  English-language,  and 
books).  If  you  set  each  limit  separately  and  redo  the  previous  search  each 
time,  FirstSearch  counts  three  additional  searches.  If  you  set  all  the  limits 
before  redoing  the  search,  FirstSearch  counts  only  one  additional  search. 

• Second  example  (1  search  counted): 

- Set  the  limits  used  in  the  first  example  and  search  for  fetal  alcohol  syndrome 
and  education  to  retrieve  the  same  final  results  as  the  first  example. 
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Vendor 

Category 

Information 

Additional 

Information 

No  search  is  counted  if  your  search  finds  no  records  and  at  least  one  word  or 
phrase  in  the  search  appears  in  no  records.  For  example,  the  following  search  is 
not  counted.  You  misspell  umpires  and  enter  the  search  baseball  and  umpires  in 
WorldCat.  Because  umpires  appears  in  no  records,  no  records  are  found  and  no 
search  is  counted. 

A search  is  counted  if  it  finds  one  or  more  records.  A search  is  also  counted  if  it 
finds  no  records  but  each  word  or  phrase  in  the  search  appears  in  one  or  more 
records.  For  example,  the  following  search  is  counted  although  it  finds  no 
records.  You  enter  the  search  baseball  and  radishes  in  WorldCat.  No  records  are 
found  because  no  record  contains  both  baseball  and  radishes.  However,  a search 
is  counted  because  baseball  appears  in  one  or  more  records  and  radishes  appears 
in  one  or  more  other  records. 

What  is  not  counted.  FirstSearch  activities,  such  as  those  listed  below,  are  not 
counted  as  searches: 

- Selecting  a database 

- Asking  FirstSearch  to  suggest  a database 

- Viewing,  printing,  e-mailing,  or  downloading  records  (Viewing  or  e-mailing 
full-text  documents  counts  as  5 searches  for  per-search  accounts.) 

- Sorting  search  results 

- Ordering  documents  through  interlibrary  loan 

- Using  help  or  searching  within  help 

- Finding  libraries  that  own  an  item 

- Looking  up  words  in  a Browse  Index  (before  clicking  on  a link  to 
begin  a search) 

- Finding  a preferred  subject  heading  or  expanding  a subject  term 

- Using  the  Expand  or  Limit  feature  available  with  lists  of  records 
(before  clicking  on  a link  in  those  features  to  begin  a search) 

Silver-Platter 

Definition 

The  number  of  queries  sent  to  the  specified  database  during  the  selected  time 
period 

Additional 

Information 

“Not  Provided” 

Table  2.  Session 


Vendor 

Category 

Information 

Academic 

Press 

(IDEAL) 

Definition 

“Not  Provided” 

Additional 

Information 

“Not  Provided” 

Bell  & 
Howell 

Definition 

“Not  Provided” 

Additional 

Information 

“Not  Provided” 

EBSCO 

Definition 

A session  is  counted  each  time  a use  logs  in. 

Additional 

Information 

The  session  will  remain  active  for  10  minutes  beyond  the  last  activity  by  the 
user.  If  the  user  returns  it  the  workstation  after  longer  than  a 10-  minute  delay, 
the  user  will  be  reconnected  and  the  session  continues  uninterrupted.  The 
session  count  will  increment  on  each  such  reconnect. 

Elsevier 

(Science 

Direct) 

Definition 

Sessions  are  defined  as  a series  of  consecutive  actions  if  one  IP  or  one  user. 

Additional 

Information 

• A session  starts  at  the  moment  of  a first  request  of  a user  and  ends  after  30 
minutes  of  no  activity. 

■ Average  session  duration  is  the  time  of  the  average  session-length  of  all  users 
of  a customer  per  month. 

■ A user  is  defined  as  a distinct  IP-number  or  as  a Registered  User  (registering 
with  Username  and  Password). 

■ (N.B.:  in  the  web-environment  it  is  impossible  to  monitor  whether  a user  has 
left  a site.  To  be  able  to  denote  when  a session  ended  the  (artificial)  30-minute 
no-activity  threshold  is  used  to  define  the  session-end  of  a user). 

Gale  Group 

Definition 

A session  is  counted  for  a particular  database  each  time  a user  enters  that 
database  within  an  overall  session  (which  began  when  they  first  logged  in  to  the 
service).  For  example,  if  a user  leaves  one  database  goes  into  another  and  then 
re-enters  the  original  database  again  during  that  same  overall  session  the  session 
count  for  the  original  database  would  be  two. 

Additional 

Information 

After  6 minutes  of  user  inactivity,  the  user  interface  (UI)  calls  the  back-end 
server  (ITS)  close  down  the  associated  session. 

Lexis-Nexis 

Definition 

“Not  Provided” 

Additional 

Information 

“Not  Provided” 

Net  Library 

Definition 

A user  session  is  defined  as  a cluster  of  web  site  usage  (time  based)  of  any 
eBook-related  web  page  on  the  Net  Library  web  site  by  an  individual.  EBook- 
related  web  pages  include  search,  bibliographic  summary  display,  checkout, 
eBook  viewing,  login,  logout  and  bookshelf  activity. 

Additional 

Information 

A user  session  cluster  is  considered  terminated  after  30  minutes  of  inactivity 

Silver  Platter 

Definition 

The  number  of  times  the  usemame(s)  logged  into  the  database  (for  example: 
MEDLINE  Advanced)  during  the  selected  time  period. 

Additional 

Information 

“Not  Provided” 
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Table3.  Items  Requested 


Vendor 

Category 

Information 

Academic 

Press 

(IDEAL) 

Definition 

“Not  Provided” 

Reported 

Statistics 

Full  text,  reference,  abstract,  table  contents,  PDF 

Bell  & 
Howell 

Definition 

“Not  Provided” 

Reported 

Statistics 

Abstracts,  citations,  text  only,  text+graphics,  page  image 

EBSCO 

Definition 

“Not  Provided” 

Reported 

Statistics 

Full  text,  reference,  abstract,  table  contents,  PDF,  citation,  e-mail,  Fax, 
Text+Graphics,  page  image,  hits,  summary  section 

Full  text-  this  statistic  increments  for  the  following  actions: 

■ The  full  display  (abstract  view)  of  a search  result  is  retrieved  and  there  is  full 
text  (full  text  is  retrieved  automatically)  Includes  links  from  result  list  or 
next/previous  records  from  full  display  where  full  text  is  displayed.  * 

■ A full  text  link  is  followed  from  the  result  list 

■ The  ASCII  full  text  is  printed,  saved  or  emailed  using  the  print/email  function. 

++ 

■ A persistent  Link  to  full  text  is  followed. 

*If  the  user  displays  the  same  record  multiple  times,  each  view  is  counted. 

**If  the  user  views  a record  then  prints  it,  the  statistics  are  incremented  for  both 
actions. 

PDFs-  this  statistic  increments  each  time  an  EBSCOhost  PDF  is: 

■ Viewed  using  regular  EBSCOhost  session* 

■ Viewed  using  persistent  links 

■ Viewed  using  SmartLinks.  Note  that  this  counts  only  EBSCOhost  PDFs  and 
will  not  count  SmartLinks  to  e Journals  managed  through  EBSCO  online 

* If  the  user  displays  the  same  PDF  multiple  times,  each  time  the  statistic  is 
incremented 

Abstracts-This  statistic  increments  for  the  following  actions: 

■The  full  display  (abstract  view)  of  a search  result  is  retrieved  (includes  links 
from  result  list  or  next/previous  records  from  full  display) 

■A  full  text  link  is  followed  from  the  result  list 

■A  citation  is  printed,  saved  or  emailed  using  the  print/email  function** 

■The  ASCII  full  text  is  printed,  saved  or  emailed  using  the  print/email  function 
(abstract  is  retrieved  to  include  with  the  full  text  display) 

■A  persistent  link  to  and  article  detail  full  text  is  followed. 

*If  the  user  displays  the  same  record  multiple  times,  each  view  is  counted. 

Elsevier 

(Science 

Direct) 

Definition 

“Not  Provided” 

Reported 

Statistics 

Full  text  articles  were  provided  in  PDF,  HTML,  PS  and  article  summary  format 
(header,  abstract,  images  and  the  list  of  reference)  e-mail,  table  of  contents, 
citation. 

■A  proper  filtering  should  remove:  All  log  records  generated  when  a page 
requests  the  embedded  image  files  and  objects  (.GIF,  .JPG,  .JPEG,  frames)  and 
all  log  records  containing  a return  code  other  than  “200”(successful  request),  or 
“301”(redirect)  or  “304”(caching)  are  filtered. 

■ articles  can  be  accessed  as  a result  of  a search  or  through  browsing  (e.g.  from 
table  of  Contents  to  Article) 
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Vendor 

Category 

Information 

Gale  Group 

Definition 

• Retrievals  - Reteievals  are  counted  for  each  article  printed  from  an  attached 
printer  or  emailed  through  the  service.  Browser  prints  are  not  counted.  The 
Journal  retrievals  ststistics  counts  articles  printed  from  an  attached  printer, 
emailed  through  the  system,  downloaded  in  PDF  or  postscript  format,  and 
articles  printed  through  the  PrintStation  in  the  “Total”  retrievals  column.  The 
journal  retrievals  ststistics  has  a separate  column  for  articles  printed  through  the 
PrintStation.  An  article  is  counted  once  persession  regardless  of  times  it  was 
retrieved.  Note  that  for  GaleNet  products,  all  retrievals  are  counted  as  “Views” 
even  if  the  article  was  formatted  for  printing  or  emailed. 

• Views-  Views  are  on-screen  displays  of  articles  from  within  a particular 
database.  These  are  counted  when  the  user  enters  a citation  (clicks  on  that 
citation  in  a search  results  list)  to  view  the  article.  An  article  is  counted  only 
once  per  session  regardless  of  the  number  of  times  it  was  viewed.  Note  that  for 
GaleNet  products,  all  retrievals  are  counted  as”  Views”  even  if  the  article  was 
formatted  for  printing  or  e-mailed. 

Reported 

Statistics 

Full  text,  abstract,  PDF,  citation,  e-mail,  Fax,  post  script  format,  print  station, 
hits 

Lexis-Nexis 

Definition 

•Documents  Retrievals  refers  to  the  number  of  document  list  pages  that  are 
retrieved  for  a specific  search  form. 

• Assume  that  access  of  specific  URL  equals  document  retrieval/search. 

• It  is  suggested  that  documents  actually  opened  be  measured. 

Reported 

Statistics 

Full  text,  document  retrievals 

netLibrary 

Definition 

Pageview  - A patron  viewed  a page  of  an  eBook  online,  using  a web  browser. 
Freepageview  - A patron  viewed  a page  of  a public,  free  eBook  online,  using  a 
web  browser. 

Checkout  - A patron  explicitly  checked  out  an  eBook,  reserving  it  for  exclusive 
use  for  a defined  period  of  time. 

Browse  - A patron  implicitly  checked  out  an  eBook  by  opening  and  viewing  it 
via  a web  browser,  by  clicking  on  the  link  “Browse  this  eBook  online”  on  the 
bibliographic  summary  page  for  the  eBook. 

Reported 

Statistics 

Pageview,  Freepageview,  Checkout,  Browse 

Silver  Platter 

Definition 

• Abstract  viewed  - The  number  of  abstracts  requested  for  display  by  the 
retrieval  software  (WebSPIRS,  WinSPIRS  or  MacSPIRS).  This  field  does  not 
necessarily  indicate  the  number  of  records  read  by  end  user. 

• Records  viewed  - The  number  of  abstracts  requested  for  display  by  the 
retrieval  software  (WebSPIRS,  WinSPIRS  or  MacSPIRS).  This  field  does  not 
necessarily  indicate  the  number  of  records  read  by  end  user. 

Reported 

Statistics 

Abstract,  Full  text 
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Table4.  Tumaways 


Vendor 

Category 

Information 

Academic 

Press 

(IDEAL) 

Definition 

“Not  Applicable” 

Additional 

Information 

“Not  Applicable” 

Bell  & 
Howell 

Definition 

“Not  Applicable” 

Additional 

Information 

To  access  this  system,  users  must  have  a valid  username  and  password 

EBSCO 

Definition 

“Not  Applicable” 

Additional 

Information 

Databases  under  this  study  do  not  fall  under  any  kind  of  simultaneous  use 
restrictions 

Elsevier 

(Science 

Direct) 

Definition 

“Not  Applicable” 

Additional 

Information 

“Not  Applicable” 

Gale  Group 

Definition 

Counted  for  each  failed  entry  into  a database  due  to  enforcement  of 
simultaneous  user  limits. 

Additional 

Information 

To  access  this  system,  users  must  have  a valid  username  and  password 

Lexis-Nexis 

Definition 

“Not  Applicable” 

Additional 

Information 

“Not  Applicable” 

NetLibrary 

Definition 

A patron  was  unable  to  view  an  eBook  for  one  of  the  following  reasons. 

■ The  library  or  consortium  doesn’t  own  a copy 

The  library  won  at  least  one  copy  but  all  copies  were  in  use 

■ The  library  doesn’t  own  a copy  and  the  consortium  owns  a copy  but  all  copies 
were  in  use 

■ The  patron  had  reached  the  checkout  limit,  expressed  in  number  of  eBooks  to 
be  checked  out  to  an  individual,  as  dictated  by  the  library 

Additional 

Information 

■ Statistics  are  reported  by  title. 

Silver  Platter 

Definition 

The  number  of  time  a user  was  denied  access  based  on  the  allowed  number  of 
simultaneous  users  to  the  specified  database  during  the  selected  time  period 

Additional 

Information 

“Not  Applicable” 

O 
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The  ARL  E-Metrics  Instructional  Module 


This  instructional  module  is  one  product  from  the  ARL  E-Metrics  project.  Combined  with  the  Data 
Collection  Manual  for  Academic  and  Research  Library  Network  Statistics  and  Performance  Measures , 
the  module  is  intended  to  assist  ARL  member  institutions  with  training  staff  about  the  importance  of 
assessing  network  services  and  the  process  of  collecting  and  analyzing  networked  statistics  and 
performance  measures.  Information  about  the  E-Metrics  project  and  access  to  various  project  reports  can 
be  found  at:  http://www.arl.org/stats/newmeas/emetrics/index.html. 

The  module  consists  of  the  following  three  Microsoft  PowerPoint™  presentations  and  accompanying 
notes: 


■ Part  1 (Importance)  is  designed  to  highlight  the  importance  of  network  measures  and  statistics 
to  research  libraries.  The  module  discusses  the  move  to  the  networked  model  of  information 
sources  and  services  at  research  libraries  and  summarizes  why  libraries  need  data  and  statistics 
regarding  networked  information  resources  and  services.  It  also  contains  an  overview  of  various 
initiatives  focusing  on  developing  network  measures. 

■ Part  2 (Preparation)  outlines  specific  plans  and  methodologies  to  prepare  and  implement  a 
process  for  collecting,  managing,  analyzing,  reporting  and  using  the  network  statistics  and 
measures.  It  contains  several  examples  of  data  analysis  and  reporting. 

■ Part  3 (Statistics)  describes  the  recommended  statistics  and  measures  for  networked  library 
services  contained  in  the  ARL  E-Metrics  Phase  II  Report.  For  each  statistic  or  measure,  it 
provides  a concise  definition,  intended  use  and  other  information  related  to  collecting  the  data. 

While  the  instructional  module  was  developed  primarily  for  training  purposes,  libraries  can  use  the 
materials  for  discussion  and  review  as  they  plan  the  measurement  and  evaluation  of  networked  services. 

The  module  was  constructed  with  a broad  audience  in  mind.  Some  libraries  might  want  to  use  all 
three  presentations  while  others  might  benefit  from  only  portions  of  them.  Libraries  should  use  the 
contents  to  best  suit  their  own  needs.  If  you  make  changes  and  would  like  to  share  your  modifications, 
please  send  the  modified  presentations  to  Martha  Kyrillidou  at  ARL  martha@arl.org  so  that  they  may 
become  available  to  other  libraries. 

Libraries  can  choose  from  a wide  range  of  options  in  terms  of  using  these  materials:  from  a one-day 
6 to  7 hour  immersion  format,  to  half-day  workshops  or  to  a series  of  1-  to  2-  hour  presentations/discussions 
over  a longer  period  of  time.  Collecting,  organizing  and  analyzing  network  information  deserves  careful 
attention.  We  hope  that  this  instructional  module  will  encourage  library  staff  to  discuss  the  issues  and  the 
processes  associated  with  measurements  of  library  networked  services,  and  that  it  will  contribute  to  using 
the  network  statistics  and  measures  effectively. 

Network  resources  and  services  are  changing  rapidly  and  reshaping  the  services  of  research  libraries. 

As  such,  the  materials  presented  in  the  instructional  module  will  need  to  be  continually  updated  and 
refined.  We  believe  the  instructional  module  and  the  manual  can  provide  that  guidance  and  help  library 
staff  take  action. 

Finally,  Amos  Lakos  of  University  of  Waterloo  Library  developed  the  presentations  together  with  the 
E-Metrics  study  team  (Jeff  Shim,  Charles  R.  McClure,  Bruce  T.  Fraser,  and  John  Carlo  Bertot).  We  also 
want  to  thank  Beth  McNeil  (University  of  Nebraska-Lincoln),  Christian  Boissonnas  (Cornell  University), 
and  Deborah  D.  Blecic  (University  of  Illinois  at  Chicago)  for  their  helpful  comments  and  suggestions  on 
earlier  versions  of  the  module. 


The  ARL  E-Metrics  Instructional 
Modules 


• Importance 

> designed  to  highlight  the  importance  of  network 
measures  and  statistics  to  research  libraries. 

• Preparation 

> outlines  specific  plans  and  methodologies  to 
prepare  and  implement 

• Recommended  Statistics  & Measures 

> Describes — in  detail — the  recommended  statistics 
and  measures  for  library-networked  services 
contained  in  the  ARL  E-Metrics  Phase  II  Report 


This  instructional  module  is  one  product  from  the  ARL  E-Metrics  project  and,  combined  with  Data 
Collection  Manual  for  Academic  and  Research  Library  Network  Statistics  and  Performance  Measures , is 
intended  to  assist  ARL  member  institutions  to  train  staff  about  the  importance  of  assessing  network  services 
and  the  process  of  collecting  and  analyzing  networked  statistics  and  performance  measures.  Information 
about  the  E-Metrics  project  and  access  to  various  project  reports  can  be  found  at: 
http://www.arl.org/stats/newmeas/emetrics/index.html. 


The  module  consists  of  the  following  three  Microsoft  PowerPoint™  presentations  and  accompanying  notes. 


§ Part  1 (Importance)  is  designed  to  highlight  the  importance  of  network  measures  and  statistics  to 
research  libraries.  The  module  discusses  the  move  to  the  networked  model  of  information  sources  and 
services  at  research  libraries  and  summaries  the  reasons  why  libraries  need  data  and  statistics  regarding 
networked  information  resources  and  services.  It  also  contains  an  overview  of  various  initiatives  focusing  on 
developing  network  measures. 


§ Part  2 (Preparation)  outlines  specific  plans  and  methodologies  to  prepare  and  implement  a process 
for  collecting,  managing,  analyzing,  reporting,  and  using  the  network  statistics  and  measures.  It  contains 
several  examples  of  data  analysis  and  reporting. 


§ Part  3 (Statistics)  describes  the  recommended  statistics  and  measures  for  library  networked  services 
contained  in  the  ARL  E-Metrics  Phase  II  Report  For  each  statistic  or  measure,  it  provides  a concise 
definition,  intended  use,  and  other  information  related  to  collecting  the  data 
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Training  Options 


> one-day  6 to  7-hour  immersion  workshop 

> half-day  workshops 

>a  series  of  1 to  2-hour  presentations  or 
discussions  over  a longer  period  of  time 

> self  study  & implementation 


While  the  instructional  module  was  developed  primarily  for  training  purposes,  libraries  can  use 
the  materials  for  discussion  and  review  as  they  plan  the  measurement  and  evaluation  of 
networked  services. 

The  module  was  constructed  with  a broad  audience  in  mind.  Some  libraries  might  want  to  use 
all  three  presentations  while  other  libraries  might  benefit  from  only  portions  of  them.  Libraries 
should  use  the  contents  to  best  suit  their  own  needs.  If  you  make  changes  and  would  like  to 
share  your  modifications,  please  send  the  modified  presentations  to  Martha  Kyrillidou  at  ARL 
martha@arl.org  so  that  they  may  become  available  to  other  libraries. 

Libraries  can  choose  from  a wide  range  of  options  in  terms  of  using  these  materials:  from  a one- 
day  6 to  7-hour  immersion  format,  to  half-day  workshops  or  to  a series  of  1 to  2-hour 
presentations/discussions  over  a longer  period  of  time.  Collecting,  organizing,  and  analyzing 
network  information  deserves  careful  development  and  having  library  staff  discuss  the  issues 
and  process  will  contribute  to  using  the  statistics  and  measures  effectively. 

Network  resources  and  services  are  changing  rapidly  and  reshaping  the  services  of  research 
libraries.  As  such,  the  materials  presented  in  the  instructional  module  will  need  to  be  continually 
updated  and  refined.  We  believe  the  instructional  module  and  the  manual  can  provide  that 
guidance  and  help  library  staff  take  action. 

Finally,  Amos  Lakos  of  University  of  Waterloo  Library  developed  the  presentations  together 
with  the  E-Metrics  study  team  (Jeff  Shim,  Charles  R.  McClure,  Bruce  T.  Fraser,  and  John  Carlo 
Bertot).  We  also  want  to  thank  three  reviewers  from  the  participating  libraries  for  their  helpful 
comments  and  suggestions  on  earlier  versions  of  the  module. 
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ARL  New  Measures  Initiative 
The  E-Metrics  Project 


The  Importance  of  Network 
Measures  and  Statistics 
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The  ARL  New  Measures  Initiative 


* Service  Effectiveness  Measures  (LibQUAL+) 

* Higher  Education  Outcomes  Research  Review 

* Investigation  of  Cost  Drivers  (i.e.  Technical 
Services) 

* Self-study  Program  Based  on  ILL/DD  Study 

* E-Metrics  (Measures  for  Electronic  Resources) 


ARL  New  Measures  Initiative 
http://www.arl.org/stats/newmeas/newmeas.html 


Working  Definition  of  Networked 
Services 

Those  electronic  information  resources  and/or 
services  that  users  access  electronically  via  a 
computer  network: 

> From  onsite  in  the  library 

> Remote  to  the  library — but  from  a 
campus  facility 

> Remote  from  the  library  & campus 
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The  working  definition  of  networked  services  is  those  electronic  information  resources 
and/or  services  that  users  access  electronically  via  a computing  network  ( l)  from  onsite  in 
the  library  (2)  remote  to  the  library,  but  from  a campus  facility,  or  (3)  remote  from  the  library 
and  campus.  Examples  of  networked  resources  include  local,  regional,  and  statewide  library 
hosted  or  authored  Web  sites  and  library- licensed  databases  (e.g.,  InfoTrac,  EBSCOHost, 
JSTOR,  Project  Muse). 

Examples  of  networked  services  include: 

Text  and  numerical  databases,  electronic  journals  and  books; 

Email,  listservs,  online  reference/assistance; 

Training  in  the  use  of  these  resources  and  services; 

Request  for  services  via  online  forms  (i.e.,  interlibrary  loans). 

The  range  and  types  of  services  accessible  through  and  supported  by  networks  will  continue 
to  evolve  as  network  technology  changes.  While  there  is  excitement  with  all  the 
developments  related  to  the  provision  of  networked  services,  there  are  a number  of  challenges 
that  require  resolution  in  the  area  of  statistics  and  measures  for  networked  services. 
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Networked  Information  Resources 


> Locally  Licensed  Databases 

> Regional  or  Statewide  Consortia 
Licensed  Databases 

> Aggregated  Databases 

> Publishers  Databases 

> Publicly  Available  (Web)  Resources 


Locally  Licensed  Databases 

These  are  databases  that  are  licensed  to  be  used  by  the  users  of  the  local  institution  only. 
Usually,  access  is  provided  to  a local  IP  range. 

Regional  or  Statewide  Consortia  Licensed  Databases 

Databases  that  are  licensed  through  a consortia  arrangement  The  consortia  may  be  based 
on  libraries  located  by  region,  type  of  Ibrary,  or  by  librariaes  that  are  part  of  a state  or 
province.  It  may  also  be  a consortia  made  up  of  libraries  belonging  to  some  organization 
(Cal  State,  etc.) 

Aggregated  Databases 

Resources  available  from  a aggregator  service  (ProQuest  - ABI  Inform) — which  puts 
together  access  to  electronic  resources  from  a variety  of  publishers.  Access  to  the 
resources  is  through  a common  search  interface,  with  additional  services  added.  Costs 
vary. 

Publishers  Databases 

Resources  available  or  restricted  to  one  publisher.  Each  publisher  will  have  its  own 
interface  and  various  types  of  delivery  formats,  etc. 

Publicly  Available  (Web)  Resources 

These  are  resources  available  freely  (mostly)  from  the  Internet.  No  need  for  particular 
lease  agreement. 
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Networked  Information  Services 


> Access  to  text  & numerical  databases, 
electronic  journals,  electronic  books,  listservs, 
e-mail 

> Instruction,  Training  & Workshops 

> Reference  & Information  Services 

> Virtual  Reference 

> Interlibrary  Loans,  Document  Delivery 

> IT  Infrastructure 

> Institutional  & Personal  Portals 


Access  to  text  & numerical  databases,  electronic  journals,  electronic  books,  listservs,  e-mail 

Clients  have  access  to  all  the  resources  made  available  through  the  library’s 
electronic  gateway.  The  institution  provides  access  to  most  electronic  resources 
and  services  to  all  registered  clients  through  some  type  of  authentifications  and 
authorization  process.  Usually  the  service  provided  is  free  to  members  - eg.,  the 
service  is  subsidized. 


Instruction,  Training  & Workshops 

Educational  services,  be  they  instructional,  training,  or  workshops,  given  to  registered  users  in  the  use 
of  the  available  electronic  resources.  These  may  be  general  in  nature,  or  related  to  the  use  of  particular 
resources  or  tools. 


Reference  & Information  Services 

Information  services — restricted  to  authorized  clients  or  to  the  general  walk-in  public — in  support  of 
information  needs.  These  may  be  basic  reference  services,  personalized  consultative  services, 
telephone  reference,  e-mail,  etc. 


Virtual  Reference 

New  type  of  reference  service  available  through  the  Web — remotely  in  real  time — usually  available 
any  time-  any  place-24/7  service.  These  services  are  evolving  and  libraries  are  investigating  new  modes 
and  techniques  in  order  to  enhance  the  potential  growing  demand  for  these  services. 

Interiibrary  Loans,  Document  Delivery 

A resource  sharing  system  is  of  particular  importance  in  order  to  round  out  the  services  that  are  not 
available  locally  or  through  local  or  regional  services  and  resources.  Consortia  sharing  arrangement 
have  to  be  developed  and  extended  through  regional,  state  and  international  systems  and  agreements 
that  work. 

IT  Infrastrnctnre 

IT  infrastructures  that  are  effective  and  reliable  are  of  particular  sensitivity.  If  the  IT  systems  are  not 
well  set  up  and  they  do  not  deliver,  the  whole  expenditure  on  good  resources  will  not  be  as  effective  to 
the  client.  IT  infrastructure  has  to  be  designed,  maintained,  and  upgraded  from  the  client  point  of 
view—  this  is  especially  a service  quality  issue. 

i 

Institutional  & Personal  Portals 

Portals  are  the  most  transformational  environment  for  effective  delivery  and  use  of  electronic 
resources.  They  are  of  strategic  importance.  Libraries  can  channel  information  where  it  is  needed 
(customization)  as  well  as  allowing  the  end  user  to  personalize  their  information  search  and  use 
environment.  Portals  have  to  be  built  as  pafbof^n Enterprise  and  total  knowledge  environment.  They 
need  to  be  planned  and  delivered  with  the  Customer  in  mind — deliver  user-side  simplicity  based  on 
complicated,  difficult  back-end.  Portals  will  also  change  the  way  libraries  and  librarians  work  in  the 
very  near  future. 
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The  Changed  Library 
Information  Environment 
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Percentages  of  Acquisitions  Dollars 
Devoted  to  Electronic  Resources 
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■ Fiscal  Year 

1 999-2000  several  libraries  reported  over  20%  spent  for  electronic  resources. 
Source:  ARL  Statistics  and  Supplementary  Statistics. 


Average  ARL  library  acquisitions  expenditures  for  electronic  resources  from  1 992/93  to 
1999/2000.  Rising  from  3.6%  (US$172,532)  in  1992/93  to  13.2%  (US$943,541)  in  1999/2000. 


Several  libraries  are  spending  over  20%  of  their  acquisition  $ on  electronic  resources  and  this 
shift  from  print  resources  to  electronic  resources  will  increase,  as  more  resources  are  being 
digitized,  archived,  and  new  cost  models  become  more  attractive.  This  also  means  that 
expenditures  on  IT  infrastructure  will  continue  to  increase  and  staffing  to  service  these  changing 
service  patterns  become  more  clear. 
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Comparison  of  Annual  Increases  in  Electronic  Resources  & 
Total  Materials  Expenditures 

ARL  Libraries  Comparison  of  Yearly  Increases  In  Electronic  Resources  and  Total  Materials 
Expenditures  Average  Counts 


■47.9% 


z~-  ; — ~ 

- „V  .t  ^ _ - 

' ■.  . ' ■ 

• 's-V 

i ■ ' 

• > . 

; r;p.. 

232% 
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1992-93  1993-94  1994-95  1995-96  1996-97  1997-98  1998-99  1999- 

2000 

■ Average  Increase  in  Bedroric  Ftesouroes  Expend lires 

■ Average  Increase  in  Ltorary  Ntete  ta I Expend tures 

> 446.9%  increase  in  dollars  spent  for  Electronic  Resources  between  1992 and  2000 

> 49. 4 % Increase  In  dollars  spent  for  Total  Library  Materials  between  1 992 and  2000 
Source:  ARL  Statistics  and  Supplementary  Statistics 


Comparison  of  average  annual  increases  in  acquisitions  expenditures  in  ARL  libraries  since  1992/93  to 
1999/2000  show  the  following: 


•The  average  increases  for  total  materials  expenditures  is  under  10%  and  is  dropping. 

•The  average  increases  for  electronic  materials  expenditures  is  always  higher. 

•Although  the  increase  during  1993-1995  seem  to  be  very  high,  they  may  be  attributed  to  the  rise  of  use  of 
CDs  at  the  time.  With  the  arrival  of  the  usable  Web  (browsing),  the  market  shifted,  but  the  increase  in 
average  expenditure  for  electronic  resources  never  went  below  15%,  and  does  not  show  any  slowdown. 
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Why  You  Need  Networked  Data  & 


Statistics 

Funding 


> Financial  Support 

> To  justify  - make  a case  for  continued 
current  support  for  digital  collections 

> To  make  a case  for  additional  support  for 
technology  & infrastructure 


The  development  of  library  networked  statistics  and  performance  measures  is  receiving  increased  attention 
and  support  There  is  a broad  recognition  for  the  need  of  network  statistics  and  performance  measures  that 

Need  to  Secure  Funding  for  the  Increased  use  ofNetworked  Resources  and  Services: 

•Assist  libraries  in  demonstrating  the  use  of  digital  collections  in  order  to  make  a case  for  continued 
collection  development  and  support;  In  a time  of  financial  constraints,  libraries  have  to  make  a case  for 
continued  levels  of  support  — no  cutbacks. 

•Assist  libraries  in  making  a strong  case  for  additional  support  of  technology  and  information  infrastructure 
by  documenting  their  Internet-based  services  and  resources;  We  need  to  demonstrate  that  IT  infrastructure  is 
essential  for  research  and  learning,  costs  are  not  dropping  and  demand  for  services  are  increasing. 
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Why  You  Need  Networked  Data  & 
Statistics 


Infrastructure 

> Better  Internal  Processes 

>To  measure  & track  changes  in 
internal  processes 

>To  enable  better  decision  making  in 
allocating  & prioritizing  resources  & 
needs 

>To  enable  assessment  of  service 
quality  in  a networked  environment 
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To  demonstrate  the  need  of  building,  maintaining,  innovating,  and  delivering  superior  services  based  on 
sound  local  and  networked  resources  and  services: 


•Allow  libraries  to  measure  and  track  internal  changes  to  library  operations  as  well  as  uses  and  users  of 
library  resources  and  services.  This  will  allow  libraries  to  measure  quantity  and  quality  of  processes  involved 
in  enabling  effective  delivery  of  networked  services.  It  will  also  allow  for  better  space  and  staff  planning. 


•Provide  a decision-making  framework  for  library  staff,  managers,  and  administrators  to  determine 
resource  allocation  strategies  and  meet  other  management  needs.  A system  designed  to  support  managerial 
decision  making  and  planning,  and  to  be  able  to  anticipate  the  future.  Will  allow  allocation  of  resources  and 
services  based  on  the  strategic  goals  of  the  institution. 


•Provide  a means  by  which  to  measure  the  quality  of  library  services  and  resources  in  the  networked 
environment.  Will  give  a better  picture  of  the  expectations  of  our  clients,  their  service  expectations  as  well  as 
how  well  the  library  is  fulfilling  those  expectations.  Will  allow  the  libraries  to  be  more  proactive,  controlling 
reliability  of  services — which  is  of  utmost  importance  in  a networked  environment. 


•Facilitate  the  expansion  from  traditional  library  use  measures  such  as  circulation,  reference  transactions, 
interlibrary  loans,  etc.,  to  include  network  measures  that  describe  the  nature  and  use  of  library-based  network 
activities  and  resources. 
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Why  You  Need  Networked  Data  & 
Statistics 

For  Comparisons 

> Institutional  Comparisons 

> For  benchmarking  digital  services 

>To  enable  competition  for 
resources  with  other  departments 
on  campus 


•Allow  libraries  to  effectively  compare  themselves  to  others  in  terms  of  Internet-based  collection  and  service 
development,  costs,  provision  of  services,  connectivity,  and  use. 


•Enable  library  directors  and  administrative  libraiy  agencies  to  compete  for  resources  with  other 
organizations  and/or  departments  by  documenting  the  range,  extent,  and  impact  of  library-provided 
networked  services. 
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Why  You  Need  Networked  Data  & 
Statistics 

Vendor  Negotiation 

> Need  for  accurate  reporting  of  network 
use 

> Need  for  accurate  estimates  of  per  client 
use 

> Ability  to  compare  overlapping  coverage 

> Need  the  ability  to  pressure  vendors  to 
price  according  to  the  library’s  real  need 


Need  for  accurate  reporting  of  network  use 

Libraries  need  accurate  reporting  of  network  use — this  can  only  be  done  if 
vendors  use  standard,  agreed  upon,  measures  of  use — as  the  activity  can  only  be 
assessed  on  the  server  side.  This  information  will  also  enable  libraries  to  plan 
expansion  of  their  networks,  network  capacity,  bandwidth,  and  speed  of 
delivery. 


Need  for  accurate  estimates  of  per  client  use 

Libraries  need  to  have  better  understanding  of  average  network  use  per  client,  in  order  to 
enable  better  hardware,  software,  staff  investment,  as  well  as  better  know  which  services 
are  in  higher  demand  than  other — this  will  help  in  network  resource  planning. 


Ability  to  compare  overlapping  coverage 

It  is  especially  important  to  be  able  to  compare  resource  overlap  between  vendors. 
Aggregators  compete  for  coverage  by  the  number  of  resources,  full-text  vs  abstracts 
only,  service  quality,  cost,  etc.  Libraries  will  have  to  be  able  to  make  acquisition  decision 
based  on  this  data. 


Need  the  ability  to  pressure  vendors  to  price  according  to  the  library’s  real  need 

The  current  price  models  are  in  constant  flux.  Libraries  at  this  time  are  disadvantaged  by 
the  conflicting  price  structures  on  the  market  Libraries  need  to  develop  the  ability  to 
price  the  networks  services  according  to  local  needs.  This  will  only  happen  if  libraries 
have  accurate  measures  of  service  from  the  various  overlapping  vendors. 
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Does  this  Describe  your  Library? 


> “We  do  not  currently  have  this  data.” 

> “We  are  seeking  ways  to  find 
measures.” 

> “We  are  looking  for  formal  mechanisms 
for  dissemination  and  evaluation.” 

> “We  are  discussing  how  to  support 
analysis  of  the  data.” 


Source:  February  2000  Scottsdale  E-Metrlcs  Planning 
Workshop 
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Changes  Needed  to  Implement  the 
E-Metrics  in  Research  Libraries 


> Need  an  Organizational  Culture  Change 

> With  Focus  on  Customers 

> With  Focus  on  Networked  Services 

* Local,  State,  Consortia,  etc. 

> Need  to  Create  a Culture  of  Assessment  e 
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Organizational  Culture  Change 

In  educational  institutions  at  all  levels  there  is  also  general  agreement  on  the  need  to  foster  collaboration  in 
achieving  institutional  missions  and  that  this  requires  a culture  of  improvement.  Assessment  within 
institutions  is  generally  seen  as  a key  lever  for  creating  an  institutional  culture  of  improvement,  inquiry, 
responsibility,  and  (in  the  language  of  some  circles)  quality. 


The  challenge  associated  with  making  assessment  more  influential  in  libraries  is  an  amalgamation  of  the 
librarian  profession’s  set  of  values  and  the  parent  oi^ganization’s  value  set.  A profession  that  sees  itself  as 
“doing  good”  is  less  concerned  with  outcomes  and  impacts,  since  it  sees  its  activities  as  inherently  positive. 
Assessment  activities  also  require  a certain  skill  set,  which  has  not  been  readily  available  to  the  profession. 
The  evolution  of  library  activities  into  functional  silos  such  as  circulation,  cataloguing,  acquisition,  reference 
service,  imposed  an  organizational  structure  that  assigned  to  the  periphery  the  activities  concerned  with  data, 
planning,  surveys,  etc.  To  change,  libraries  have  to  incorporate  assessment  into  their  everyday  activities, 
they  have  to  create  structures  for  assessment  activities  and  use  these  measures  to  create  environments  that  are 
effective  and  truly  client  centred. 


To  focus  on  client  needs,  libraries  must  base  their  services  on  the  expressed  needs  and  requirements  of  their 
clientele,  to  deliver  high  quality  service,  and  to  find  ways  to  ensure  service  quality.  The  focus  on  results,  on 
outcomes,  on  added  value  is  essential.  These  are  the  prerequisites  for  creating  a culture  of  assessment 
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Culture  of  Assessment 
Definition 


> Basic  Value -customer  & teaming  focus 

> A Culture  of  Assessment  is  an  organizational 
environment  in  which 

> decisions  are  based  on  facts,  research,  and  analysis, 
and  where 

> services  are  planned  and  delivered  in  ways  that 

> maximize  positive  outcomes  and  impacts  for  library 
clients. 

> A Cult  ire  of  Assessment  exists  in  organizations  where 

> staff  care  to  know  what  results  they  produce  and 

> how  those  results  relate  to  customer  expectations. 

> Organizational  mission,  values,  structures,  and  systems 
support  behavior  that  is  performance  and  learning 
focused. 
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Defining  a "Culture  of  Assessment" 

Definition: 

A Culture  of  Assessment  is  an  organizational  environment  in  which  decisions  are  based  on  facts, 
research,  and  analysis,  and  where  services  are  planned  and  delivered  in  ways  that  maximize 
positive  outcomes  and  impacts  for  customers  and  stakeholders.  A Culture  of  Assessment  exists  in 
organizations  where  staff  care  to  know  what  results  they  produce  and  how  those  results  relate  to 
customers'  expectations.  Organizational  mission,  values,  structures,  and  systems  support  behavior 
that  is  performance  and  learning  focused. 


A Culture  of  Assessment  exists  when: 

• The  library's  mission,  planning,  and  policies  are  focused  on  supporting  the  customer's 
information  and  communication  needs 

•Performance  Measures  are  included  in  library  planning  documents  such  as  strategic  plans 
•Library  administrators  are  committed  to  supporting  assessment 

•Staff  and  leaders  recognize  the  value  of  assessment  and  support  and  participate  in  assessment  as 
part  of  their  regular  assignments 

•Individual  and  organizational  responsibility  for  assessment  is  addressed  explicitly 

•Continuous  communication  with  customers  is  maintained  through  needs  assessment,  quality 
outcome  and  satisfaction  measurements 

•Relevant  data  and  user  feedback  is  routinely  collected,  analyzed,  and  used  to  set  priorities, 
allocate  resources,  and  make  decisions 

•Assessment  activities  can  be  supported  by  a Management  Information  System  or  Decision 
Support  System 

•All  library  services,  programs,  and  products  are  evaluated  for  quality  and  impact 
•Staff  continuously  improve  their  capability  to  serve  customers  and  are  rewarded  for  this 
•Units  and  staff  have  customer  focused  S*M*A*R*T*  goals  which  are  monitored  regularly 
•Rewards  support  removing  barriers  to  quality  customer  service 
•On-going  staff  development  in  the  area  of  assessment  is  provided  and  supported 
•Staff  appreciate  feedback  and  support  for  achievement  of  performance  and  learning  goals 

Originally  developed  by  Amos  Lakos  (University  of  Waterloo)  and  Betsy  Wilson  (University  of 
Washington)  - 1998 

Revised  and  updated  by  Amos  Lakos  and  Shelley  Phipps  (University  of  Arizona)  - LTFEU-  2000 
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Culture  of  Assessment 

We  have  a GAP 


Espoused  Values  & Needs: 
Recognition  of  Importance  of  Assessment, 
Client  Centered  Procedures,  Allocations 


GAP 


Actual  Situation: 

No  support  & rewards  system;  lacking 
Infrastructure:  MIS,  skills,  training 


Applying  the  Culture  of  Assessment  IQ  Test  in  a number  of  ARL  workshops  and  through  the  Service 
Quality  Online  Lyceum  show  that  libraries  have  a gap  between  their  understanding  for  their  needs  for  doing 
more  assessment  in  a systematic  way  and  the  actual  actions  they  take. 


In  general,  library  leaders  and  librarians  seem  to  recognize  the  importance  of  assessment,  the  need  for 
developing  policies  and  procedures  to  meet  clients  needs,  to  consider  client  needs  in  allocating  resources. 


However, 

There  is  in  general  low  level  of  support  and  reward  mechanisms  for  staff  engaged  in  assessment,  there  is  a 
lack  of  infrastructure  to  support  assessment  work,  almost  no  MIS  in  libraries,  and  not  enough  training  and 
development  of  assessment  based  skills. 
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Bridging  the  Gap 


> Think  Systemicaily  (SIPOC) 

> Anticipate  the  Future 

> Take  Risks  - innovate  & experiment 

> Focus  on  Customers 

> Create  & Sustain  a Culture  of 
Assessment 


Systems  Thinking 

Based  on  the  SIPOC  model:  suppliers,  inputs,  processes,  outputs,  customers,  (see  Peter  Scholtes,  The  Leader’s  Handbook.) 

Creating  an  assessment  environment  and  the  corresponding  MIS  infrastructure  depends  on  seeing  the  whole  picture,  its 
various  components,  and  the  links  between  them.  Administration  and  staff  have  to  be  encouraged  to  look  beyond  the 
details.  Understanding  organizational  purpose,  seeing  the  big  picture,  being  customer-centered,  and  understanding  the 
links  and  interrelations  between  goals,  outcomes,  processes,  and  constant  change  are  of  primary  importance. 


Anticipate  the  Future 

It  is  important  to  be  focused  on  the  future,  both  from  a strategic  perspective  and  especially  because  e-metrics  and  network 
services  are  new  and  ever  changing — and  in  ways  we  may  not  even  envision.  Which  makes  it  even  more  important  to  be 
future  focused 


Take  Risks  - innovate  & experiment 

Old  processes  and  services  should  be  phased  out  in  order  to  focus  on  what  is  needed  strategically.  Since  assessment  work 
may  be  new  and  unfamiliar,  and  viewed  as  risky,  risk  should  be  encouraged.  In  essence,  risk  taking  will  have  to  be  the 
norm  in  libraries. 


Focus  on  Customers 

It  is  essential  to  think  from  the  perspective  of  the  customer.  Libraries  do  not  exist  for  librarians,  but  for  the  users  of 
libraries  and  for  people  who  are  looking  for  education  and  information.  Librarians  have  to  value  customer  feedback  and 
take  the  feedback  seriously — they  have  to  care  about  what  their  customers  need. 


Create  & Sustain  a Culture  of  Assessment 

Culture  of  assessment  is  essential  to  maintaining  libraries  as  relevant  institutions  in  the  new  information  environment.  The 
culture  of  assessment  pushes  the  organization  forward  toward  focusing  on  customers  and  outcomes  for  customers.  It 
encourages  self-examination  and  openness  between  staff,  customers,  and  other  stakeholders.  It  becomes  embedded  in 
everyday  processes  and  it  is  essential  for  dynamic  organizational  change.  This  in  essence  is  one  of  the  prerequisites  to 
change  that  over  time  becomes  accepted  and  changes  the  culture  of  the  organization. 


A culture  of  assessment  is  about  learning  how  to  learn.  It  is  about  developing  the  organization’s  and  the  individual’s 
learning  capabilities.  It  necessitates  curiosity.  The  new  competence,  experience,  and  learning  agility  that  is  part  of  the 
creation  of  a culture  of  assessment  leads  to  new  confidence  and  enhanced  expertise.  This  is  turn  leads  to  more 
effectiveness  and  more  measurable  outcomes  and  impacts  for  customers  and  stakeholders. 


Organizational  culture  change  still  needs  an  amalgam  of  committed  leadership,  repeated  articulation  of  purpose,  time,  and 
group  learning  to  really  penetrate  the  organization  and  take  root.  In  this  way,  focus  on  achieving  positive  and  tangible 
outcomes  assisted  by  a culture  o fas sessmen twill  contribute  to  positive  organizational  culture  change,  but  may  not  be 
decisive  by  itself.  Assessment  has  to  become  part  of  the  work  processes,  part  of  the  organizational  structure,  part  of 
organizational  learning,  and  part  of  the  decision-making  loop,  in  order  to  impact  the  culture  and  act  as  a catalyst  for 
organizational  change.  Culture  of  assessment  has  to  become  a basic  value  to  the  organization,  it  has  to  become  embedded 
in  everyday  work,  automatic,  taken  for  granted — it  has  to  become  our  culture. 
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Bridging  the  Gap 


> Build  Organizational  Structures  that 
Support  Work  based  on  Data  & Analysis 
- MIS  Resource 

> Invest  in  Staff  Resources  & Learning 

> Invest  in  IT  Infrastructure,  Services  & 
Support 

> Build  Collaborative  Environments  with 
Vendors  & Other  Libraries 


Build  Organizational  Structures  that  Support  Work  based  on  Data  & Analysis  - MIS  Resource 

A Management  Information  Service  is  set  up  in  order  to  support  the  data  and  information  needs  of  the  organization. 

Library  leaders  who  possess  clearly  defined  expectations,  and  understand  the  need  for  data  and  information  to  support 
dec  ision  making,  will  support  the  MIS.  The  MIS  unit  will  be  responsible  for  the  coordination  of  all  assessment  activities, 
identification  of  information  needs,  creation  of  an  appropriate  environment  for  organizing  data  and  information,  analysis  of 
information,  and  making  information  available  to  the  processing  and  management  units  of  the  library. 


The  need  for  management  information  systems  in  libraries  was  recognised  over  twenty  years  ago.  One  of  the  earliest  and 
most  persistent  promoters  of  the  necessity  and  advantages  of  MIS  in  libraries  is  Charles  McClure  who  wrote  about  this 
already  in  the  early  1980s.  (McClure,  1980).  However,  setting  up  an  MIS  or  a DSS  requires  awareness,  commitment,  and 
resources.  Current  examples  of  functioning  MIS  systems  in  academic  libraries  are  almost  nonexistent.  Setting  up  an  MIS 
is  not  simple,  but  it  will  have  benefits,  especially  as  demands  for  accountability  are  increasing.  A MIS  will  enhance  the 
creation  of  assessment  culture  in  libraries.  In  order  to  sustain  quality  services,  institutions  that  aspire  to  be  continuously 
effective  and  successful  have  to  rely  on  decision  support  systems.  An  example  of  MIS  development  is  work  at  the 
University  of  Waterloo  Library  and  the  Tri-University  Library  Group  Consortia  between  1993-1999  (Lakos,  1998)  and 
newer  MIS  type  implementations  at  the  University  of  Virginia  and  the  University  of  Pennsylvania  Libraries. 


Invest  in  Staff  Resources  & Learning 

Staff  development  in  all  areas  of  assessment  is  needed.  This  includes  training  on  appropriate  information  analysis  tools 
and  software  as  well  as  continuous  skills  upgrading  in  assessment  work.  Many  librarians  feel  that  they  lack  the  technical 
and  computer  skills  needed,  and  resist  acquiring  those  skills.  Many  feel  that  they  lack  the  skills  to  use  data  gathering 
techniques  such  as  surveys  and  focus  groups.  They  also  are  unsure  about  using  statistics.  Professionals  hesitate  working 
in  areas  where  they  lack  knowledge,  as  this  seems  to  signify  lack  of  control.  Lack  of  skills  also  creates  lack  of  confidence. 
Much  assessment  is  not  carried  out  because  staff  lacks  the  confidence  to  tryout  new  and  unfamiliar  activities.  The  skills 
issue  may  also  be  aggravated  by  a lack  of  well- organized  technical  support  for  non-technical  staff.  Without  this  supportive 
environment,  staffs  capability  to  use  assessment  and  analysis  tools  is  derailed. 


Invest  in  IT  Infrastructure,  Services  & Support 

Libraries  have  to  plan  in  building  reliable  IT  infrastructures,  with  superior  staff  resources  and  well  integrated  services.  IT 
overall  has  to  be  developed  from  a service  perspective — which  means  from  a customer  perspective.  Libraries  should 
understand  the  importance  of  IT  for  the  delivery  of  networked  services,  but  these  services  have  to  be  part  of  the  larger 
picture — they  should  support  and  enable — not  drive. 


Build  Collaborative  Environments  with  Vendors  & Other  Libraries 

A superior  network  service  environment  cannot  be  developed  in  isolation.  Local  resources  are  not  effective  in  delivering 
information  and  services  in  the  new  reality.  For  this  reason,  developing  relationships  with  both  vendors  and  other  libraries 
is  essential.  Libraries  understand  the  need  for  managing  resources  and  services  in  a new — effective — cooperative  mode. 
They  also  understand  that  cooperation  (in  the  form  of  consortia)  or  other  modes  of  collaboration  can  be  used  to  leverage 
better  service  deals  with  vendors,  as  well  as  deliver  better  services  to  their  local  clients. 
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Additional  Approaches  to 
Evaluation 


> Balanced  Scorecard  - http://www.b3cd.com/ 

> LibQual  Project  - http^iVwwjrl.orfl/libqual 

> Activity  Based  Costing  - 

http://www.mamag.com/winter01/w01tati.htm 

> Learning  Outcomes  Assessment  - 

http://www.ar1.orct/BUt8/ngwmcaB/OLacome3/heahtml 

> Measuring  Library  Service  Quality  - ARL  Online 

Lyceum  - http://www.ar1.org/training/quality%5Fold.html 

> Measuring  Performance  of  ILL7DD  - 

http://www.ar1.org/stats/newmeas/ill-dd.html 


Additional  Approaches  to  Evaluation 
Balanced  Scorecard  - http://www.bscol.coin/ 

The  Balanced  Scorecard  developed  by  Robert  Kaplan  and  David  Norton  describes  a strategically  oriented  set  of  performance  indicators  that  are  grouped  into  four 
perspectives:  financial,  customer,  internal  proceses,  and  learning  and  growth  The  idea  is  to  link  performance  measurements  to  strategy.  The  balanced  scorecard 
gives  managers  a framework  of  integrating  and  coordinating  their  activities  and  linking  their  strategies  to  performance  metrics  and  ties  them  to  compensation 
systems  in  a meaningful  way.  This  also  helps  the  development  of  non-financial  measures  and  assigns  to  them  measurable  values.  Over  time,  there  were  many 
diverse  implementations  of  the  balanced  scorecard  and  it  was  also  adapted  to  the  non-profit  sector.  Adapting  the  balanced  scorecard  to  the  library  environment  is 
possible  and  will  tie  cur  strategic  piaposes  to  our  processes  and  rewards  and  give  a coherent  framework  to  cur  assessment  endeavors. 


LibQual  Project  - http://www.arl.org/1ibqual 

LibQUAL+  is  a research  and  development  project  undertaken  by  ARL  in  collaboration  with  Texas  A&M  University  and  with  financial  support  from  the  U S. 
Department  of  Education's  Fund  for  the  Improvement  of  Postsecondary  Education  (FIPSE)  through  September  2003.  LibQUAL+  is  defining  and  measuring  library 
service  quality  across  institutions  and  creating  useful  quality -assessment  tools  for  libraries;  it  is  one  of  the  ARL  New  Measures  Initialise  projects,  which  seek  to 
develop  innwative  ways  for  libraries  to  describe  their  contributions  to  their  institutions.  The  goals  of  LibQUAL+  are: 

•establidi  a library  service  quality  assessment  program  at  ARL; 

•develop  Web-based  tools  for  assessing  library  service  quality, 


•develop  mechanisms  and  protocols  for  evaluating  libraries;  and 
•identify  best  practices  in  providing  library  service. 

Service  quality  has  always  been  a value  for  libraries— LibQUAL+  provides  a measure  of  that  value.  LibQUAL+  currently  tests  a tool  for  measuring  library  users' 
perceptions  of  service  quality  and  identifies  gaps  between  desired,  perceived,  and  minimum  expectations  of  service.  The  project  will  continue  as  an  R&D  endeavor 
through  2003,  by  which  time  it  is  anticipated  that  LibQUAL+  will  evolve  into  an  ongoing  service  quality  assessment  program  at  ARL. 


Activity  Based  Costing  - http://www.mamag.com/winter01/w01tati.htm 

Higher  Education  Institutions  (HEIs)  are  in  a state  of  turmoil  and  fiscal  crisis.  Escalating  costs,  diminishing  resources,  increased  competition,  unhappy  customers 
(studerls,  parents),  and  state  legislators  demanding  accountability  are  pressuring  them  to  manage  their  costs  better.  These  pressires  are  not  much  different  from 
those  experienced  by  the  manufacturing  indistry  of  yesterday.  Using  tods  and  techniques  such  as  activity- based  costing  (ABC),  business  process  reengineering, 
concurrent  engineering,  and  total  quality  management,  manufacturers  gained  a better  understanding  of  costs,  simplified  products  and  procedures,  eliminated  waste, 
cut  costs,  reduced  lead  times,  improved  quality,  added  value,  and  gained  customer  satisfaction  and  loyalty.  Though  activity- based  costing  is  used  successfully  by  a 
variety  of  manufacturing  and  service  industries,  educational  institutions  are  lagging  Only  a few  colleges  and  universities  in  the  United  States  apply  ABC.  In  Great 
Britain  several  universities  have  recently  introduced  ABC  accounting.  These  universities  found  that  ABC  helped  them  with  tighter  financial  management  and  better 
resource  allocation. 


Learning  Outcomes  Assessment  - http://www.arl.org/5tats/newineas/outcoincs/heo.htnil 

The  goal  for  this  project  is  to  col  lab  ora  lively  develop  a strategy  for  involving  research  libraries  in  campus  assessment  activities  and  to  demonstrate  the  value  of  the 
library  to  the  learning  community.  Interested  ARL  directors  are  invited  to  assist  in  reviewing  higher  education  outcomes  and  defining  a process  for  libraries  to 
develop  learning  and  research  outcomes  within  their  parent  institution  context  The  HEO  project  is  an  overarching  activity  in  which  some  cf  the  more  specific 
measures  projects  can  be  placed.  Current  activities  are  supported  with  funding  from  interested  ARL  libraries. 


Measuring  Library  Service  Quality  - ARL  Online  Lyceum  - http://www.arl.org/training/quality%5Fold.htmI 

The  course  will  cover  the  background  and  theory  cf  measuring  service  quality,  methods  to  assess  and  improve  service,  and  the  impact  of  measuring  service  quality 
on  overall  library  value  to  constituencies.  A case  study  approach  to  problem  solving  will  provide  the  content  and  context  for  developing  and  understanding 
measurement  techniques. 

In  this  cou-se  you  will  learn: 

•what  measurement  of  service  quality  means; 

•theoretical  basis  for  measurement  of  service  quality; 

• practical  methods  for  measuring  service  quality,  and 
•how  to  apply  theory  and  methods  in  the  local  setting 

This  Online  Lyceum  Collaborative  Learning  Event  will  incorporate  elemeris  of  both  synchronous  (real-time)  and  asynchronous  interaction  with  course  facilitators 
and  a global  peer  network  cf  up  to  thirty  learners  via  a course  bulletin  board,  chat  rooms,  and  regular  email. 


Measuring  Performance  of  I LL/D  D - http://www.arl.org/stats/newmeas/ill-dd.html 

This  effort  is  in  development  by  Mary  E.  Jackson,  ARL  Senior  Program  Officer  for  Access  & Delivery  Services.  She  is  utilizing  the  lessons  learned  from  the 
ILL/DD  Performance  Measures  cost  study  and  the  experience  gained  from  the  worktops  on  organizational  re-design  to  improve  internal  workflows  and  procedures 
followed  in  interlibrary  lean  and  document  delivery  departments. 
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BEST  COPY  AVAILABLE 


Projects  Related  to 
E-Metrics 


> EQUINOX  Project  (EU-UK) 

> Publishing  and  Library  Solutions 
Committee  (PALS)  - (UK) 

> International  Coalition  of  Library 
Consortia  (ICOLC) 

> National  Commission  of  Libraries  and 
Information  Science  (NCLIS) 
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•European  Commission  EQUINOX  Project 

The  Equinox  project,  funded  by  the  European  Union,  primarily  focuses  on  developing  performance  measures  for 
networked  services  and  resources  in  European  academic  libraries.  This  project  continues,  but  has  shifted  from 
performance  indicators  to  software  development  to  assist  libraries  to  assess  their  network-based  services  (see 
http://equinox.dcu.ie/). 


•Publishing  and  Library  Solutions  Committee  (PALS)  Working  Group  on  Online  Vendor  Usage  Statistics  (UK) 

The  Publishing  and  Library  Solutions  Committee  (PALS)  Working  Group  on  Online  Vendor  Usage  Statistics,  established 
in  the  U.K.  and  chaired  by  Richard  Gedye,  Journals  Sales  and  Marketing  Director,  Oxford  University  Press,  will  address 
the  following; 

•Research  current  and  planned  availability  of  vendor- based  usage  statistics  for  online  products. 

•Research  current  initiatives  to  develop  accepted  codes  of  practice/guidelines  in  this  area. 

•Research  current  library  wants. 

•Produce  realistic  code  of  practice/guidelines. 

•Market  the  code  of  practice/guidelines  to  vendors  and  hosting  systems;  get  them  accepted/adhered  to. 

•Research  the  possibility  of  centralized  provision,  e.g.,  a usage  statistics  clearinghouse. 


•International  Coalition  of  Library  Consortia  (ICOLC)  review  of  ICOLC  Guidelines  for  Statistical  Measures  of 
Usage  of  Web-based  Indexed,  Abstracted,  and  Full-Text  Resources. 

Since  the  mid-90s,  an  international  coalition  of  libraries — predominantly  academic — have  been  working  towards  a 
standard  set  of  definitions  for  online  database  services.  Current  definitions  reflect  work  completed  in  December  2001  (see 
http://www.librarv.yale.edu/  consortia/web  stats.html). 


•National  Commission  of  Libraries  and  Information  Science  (NCLIS)  project  to  standardize  online  database  usage 
statistics  and  reporting  mechanisms  (public  libraries). 


NCLIS  works  cooperatively  with  NCES  in  implementing  the  Library  Statistics  Cooperative  Program.  NCLIS  serves  as  a 
liaison  to  the  library  community,  organizes  meetings  and  training  workshops,  organizes  training  and  technical  assistance, 
monitors  trends,  and  advises  NCES  on  policy  matters. 
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Projects  Related  to 
E-Metrics 


> Institute  of  Museum  and  Library 
Services  (IMLS) 

> Council  on  Library  and  Information 
Resources  (CLIR) 

> NISO  Standard  on  Library  Statistics. 

> Round-up  of  other  E-Metrics 
Developments-ARL 


•Institute  of  Museum  and  Library  Services  (IMLS)  project  to  develop  uatioual  network  online  statistics  and 
performance  measures  for  public  libraries. 

•IMLS  sponsored  the  researchers  to  develop  national  network  statistics  and  performance  measures  for  public 
libraries.  That  project  resulted  in  a network  statistics  manual  for  publ  ic  libraries  (Bertot,  McClure,  and  Ryan, 
2000).  This  work  continues  with  renewed  sponsorship  by  IMLS  and  NCLIS  to  develop  a national  data 
collection  system  for  public  library  network  statistics  (see  http://www.ii.feu.edu).  [Note:  researcher 
involvement]. 

•Council  on  Library  and  Information  Resources  (CLIR)  report  by  consultant  Judy  Luther  related  to  network 
statistics. 

•CLIR  has  engaged  consultants  to  review  the  state  of  the  art  of  network  statistics,  analysis,  and  presentation  in 
the  academic  library  and  consortia  environments.  The  report  is  expected  in  fall  2000,  and  there  is  an  indication 
that  this  work  will  continue. 

•NISO  forum  on  performance  measures  and  statistics  for  libraries  in  preparation  to  evaluate  the  NISO  Standard 
on  Library  Statistics 

•The  National  Information  Standards  Organization  (NISO)  is  undertaking  a review  and  update  of  its  z39.7 - 
Library  Statistics  standard.  This  review  and  update  will  consider  network  services  and  resources  statistics  and 
performance  measures.  As  of  October  2000,  the  planning  committee  for  the  standard  is  just  forming  and 
beginning  to  meet  to  develop  the  review  process  (see  http://www.niso.org). 

Round-up  of  Other  E-Metrics  Developments — ARL 

Some  are  already  mentioned  above,  but  we  can  add  the  following: 

DLF  Initiative 

The  Digital  Library  Federation  (DLF)  named  Denise  Troll,  Assistant  University  Librarian  for  Library 
Information  Technology  at  Carnegie  Mellon  University  Libraries,  a DLF  Distinguished  Fellow  to  spearhead  the 
part  of  the  DLF’s  program  that  aims  to  identify  and  evaluate  measures  that  are  appropriate  for  assessing  the  use 
and  effectiveness  ofdigital  library  collections  and  services.  For  more  information,  see 
<http://www.clir.org/diglib/use.htm>. 


Recent  Publications 

Statistics  and  Performance  Measures  for  Public  Library  Networked  Services,  by  John  Carlo  Bertot,  Charles  R. 
McClure,  and  Joe  Ryan.  Chicago:  American  Library  Association,  October  2000.  This  book  recommends  13 
national  statistics  and  measures  for  public  libraries. 


Performance  Measures  for  Federal  Agency  Websites,  by  Charles  R.  McClure,  J.  Timothy  Sprehe,and  Kristin 
Eschenfelder.  Washington:  U.S.  Government  Printing  Office,  October  2000.  This  report  analyzes  the  impact  of 
federal  policies  affecting  Web  site  development  and  proposes  17  performance  measures. 


White  Paper  on  Electronic  Journal  Usage  Statistics,  by  Judy  Luther.  Washington:  Council  on  Library  and 
Information  Resources,  October  2000.  This  white  paper  calls  for  working  with  publishers  to  facilitate  the 
development  of  statistics  in  the  industry.  See  <http://www.clir.org/pubs/reports/pub94/contents.html>. 
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Keep  in  Mind... 


> The  Evolving  Nature  of  the  IT  Environment 

> The  Evolving  Nature  of  Digital  Offerings — New 
Vendor  Configurations 

> New  Partnership  Arrangements 

> New  Collaborative  Service  Options — influence 
on  E-Metrics 

> New  Usage  & Payment  Models 


The  Evolving  Nature  of  the  IT  Environment 

The  IT  environment  is  constantly  changing  and  the  rate  of  change  is  very  rapid  The  IT  environment  is 
moving  toward  the  wireless,  Web-based  content  and  services  and  is  becoming  increasingly  interactive  and 
based  on  multimedia.  The  cost  of  upgrading  to  new  infrastructure  has  to  be  planned. 


The  Evolving  Nature  of  Digital  Offerings — New  Vendor  Configurations 

There  is  always  new  content  developed  and  being  packaged  and  repackaged.  New  search  modules  are 
constantly  being  developed  The  business  models  are  in  constant  state  of  change  and  new  vendor 
configurations  are  always  emerging.  Some  players  will  change,  disappear,  morph,  and  will  be  consolidated. 


New  Partnership  Arrangements 

The  market  changes  constantly  and  vendors  as  well  as  libraries  are  always  looking  for  new  partnership 
models  in  order  to  deliver  services  more  effectively  and  in  order  to  be  more  cost  effective. 


New  Collaborative  Service  Options — influence  on  E-Metrics 

Vendor  consolidation,  publishing  consolidation,  as  well  as  new  technological  advancements  will  mean  new 
service  options — usually  better  options.  Libraries  have  to  be  aware  of  these  changes  and  act  in  order  to  take 
advantage  of  them.  The  new  models  will  have  to  be  integrated  in  such  a way  that  the  E-Metrics  measures 
will  become  more  current  and  easier  to  manage. 


New  Usage  & Payment  Models 

With  better  ways  to  capture  and  analyze  usage,  information  will  be  available — and  may  be  better  used  to 
develop  more  realistic  costing  models.  The  idea  is  to  move  to  the  pay  as  you  go  model. 
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Forces  that  Shape  the  Network  Services 
Environment  in  Research  Libraries 

Internal 

> Increasing  Demand  for  Digital  Content 

> IT  Infrastructure  - Networked 

> Competencies  - Expert  Staff  (HR) 

> Accountability  - Assessment 

> Organizational  Structure 

> Institutional  Mission 

u 

Content  is  Digital 

Supply  - phenomena]  increase 
Demand  for  Services  - increases 
Persistent  Remote  Access 

* New  Business  Models  - serial  suites,  pay  as  you  go 
Competition  from  Alternative  Information  Services  - Questia,  etc. 

Future  u Digital 

•In  the  future  most  information  will  be  digital.  In  a digital  environment  it  will  be  important  to  assess  this  environment  in  order  to  exist  in  it. 

IT  Networks  Everywhere 

•Everything  - information,  communication,  transactions  will  be  interconnected  Information  services  will  all  be  digital  — therefore  libraries  need  to 
survive  as  part  of  a new  networked  environment  and  stay  relevant 
Society  demands  information  literate  and  skilled  workers. 

•Education  institutions  will  be  pressured  to  produce  graduates  who  are  skilled  in  the  digital  environment — this  will  be  one  of  the  most  measured 
outcomes. 

Competition  from  Alternative  Information  Services 

•Libraries  will  have  much  more  intense  competition  from  other  information  services.  Depending  on  demand,  efficiency  and  effectiveness 


IT  Infrastructure  - networked 

IT  Environment -evolving  & unstable 

Local  IT  Infrastructure  - evolving,  unstable  & expensive 

Portals  - Transformational  Choice 

Wireless  & Portable  Devices 

Competencies  (HR) 

Visionary  Leadership 

Library  Staff  - Expertise  + Continuous  Learning 

IT  Expertise  - retention  & training 

Staff  for  Infrastructure  Support  - MIS  & Systems 


Accountability 

•Demanded  by  Stakeholders,  Accreditation  Agencies  ^ 

•Society  Demands  Information  Competent  or  Skilled  Workers  (learning  outcomes) 

•Need  for  Understands  Assessment  (outcomes  & impacts) 

•Need  to  Demonstrate  Value  & Service  Quality. 

• Increasing  demand  for  libraries  to  demonstrate  outcomes/impacts  in  areas  important  to  the  institution. 

• Increasing  pressure  to  maximize  use  of  resoirces  - bench  mark  best  practices  to  save  or  reallocate  resources. 

Organizational  Structure 

No  functional  silos 

Everything  Is  interconnected  - linked 

Flatter  organization 

Customer  focused 

Understands  the  need  for  assessment  - closingthe  assessment  loop 

Institutional  Mission 

Articulated  Purpose 
Positive  Leadership 
Externally  Focused 
Customer  Focused 

Processes,  Staffing  & Rewards  are  tied  to  strategic  goals 
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Forces  that  Shape  the  Network  Services 
Environment  in  Research  Libraries 


External 


> Increasing  Offering  of  Digital  Content 

> Inter-institutional  Competition 

> Increasing  Ability  for  Library  Users  to 
Bypass  the  Library  for  Scholarly 
Information 

> Ever-changing  Information 
Technology  Environment 


•WJ 


Increasing  Offering  of  Digital  Content 

Much  more  digital  content  - no  stop  to  this  at  all 

Inter-institutional  Competition 

Continued  inter- institutional  competition  for  students  & resources. 

Increasing  Ability  for  Library  Users  to  Bypass  the  Library  for  Scholarly  Information 

New  content,  new  devices  and  services,  and  new  businesses  that  will  be  able  to  create 
information  services  that  work  at  a realistic  price  will  enable  clients  to  bypass  libraries. 

Ever-changing  Information  Technology  Environment 

IT  will  continue  to  change  at  much  faster  rate  than  institutions  will  be  able  to  absorb  it. 
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Take  Action... Now!!! 


> PLAN  & PRIORITIZE 

> You  Don’t  Have  to  Collect  ALL  the 
Measures 

> Collect  the  MOST  IMPORTANT  Measures 
for  Your  Library 

> EXPERIMENTATION  is  OK! 

> Take  ACTION  on  What  You  Measure! !! 
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PLAN  & PRIORITIZE 

To  be  effective  in  this  rapidly  change  environment  libraries  have  to  plan  and  make  priorities  and 
they  will  only  be  able  to  do  this  if  they  have  the  right  informatioa 


You  Don’t  Have  to  Collect  ALL  the  Measures 

Libraries  don’t  have  to  measure  all  at  once.  Collect  what  you  can,  learn,  add  resources  as  you 
learn. 


Collect  the  MOST  IMPORTANT  Measures  for  Your  Library 

Identify  and  collect  what  is  important  to  you  locally.  Identify  what  it  is  and  focus  on  it  first. 


EXPERIMENTATION  is  OK! 

It  is  OK  to  experiment.  It  is  OK  to  take  chances.  You  learn  by  experimenting.  E-Metrics  is  new. 
We  are  all  experimenting. 


Take  ACTION  on  What  You  Measure!!! 

Implement  changes  based  on  what  you  learn  from  your  measurement  activities.  Your  staff  and 
stakeholders  want  to  see  that  your  actions  have  results. 


28  4 


o 


25 


Developing  measures  and  evaluation  techniques 
for  networked  services  will  take  time,  effort, 
and  on-going  learning  on  everyone’s  part  - but 
we  must  begin  now.  _ 

9 (Carla  Stome, 

University  of  Arizona) 

We  not  only  need  to  measure  things  in  new 
ways  but  we  also  need  to  measure  new  things. 

(Sherrie  Schmidt, 

Arizona  State  University) 
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ARL  New  Measures  Initiative 
The  E-Metrics  Project 


Preparing  YOUR  Library  to 
Collect,  Report,  Analyze,  and 
Use  the  Statistics  and 
Measures 
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Effective  Assessment  Methods  & 
Procedures  Are: 

> Structured,  systematic,  and  ongoing — not 
episodic 

> Related  to  other  institutional  strategic  long- 
range  plans  and  to  planning  and  budgeting 
processes 

> Emerged  from  and  sustained  by  staff, 
faculty,  and  administrative  commitment 


Effective  Assessment  Methods  & Procedures  Are 


Structured,  systematic,  and  ongoing — not  episodic 

Assessment  activities  have  to  be  ongoing — have  to  take  place  over  time — and  they  have 
to  be  systematic.  They  heed  structure  to  be  able  to  be  sustained  over  time.  Only  over 
time  will  assessment  show  the  big  picture. 


Related  to  other  institutional  strategic  long-range  plans  and  to  planning  and  budgeting 
processes 

Assessment  should  relate  directly  to  institutional  strategic  plan,  if  there  is  one.  Any 
assessment  should  be  part  of  a planning  framework.  This  will  help  integrate  the 
assessment  activity  into  the  institutional  processes  and  outcomes. 


Emerged  from  and  sustained  by  staff,  faculty  and  administrative  commitment 

Assessment  work  should  be  developed  systemically  by  involving  staff  at  all  levels.  This 
way  staff  will  understand  the  purpose  behind  the  assessment  effort. 
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Effective  Assessment  Methods  & 
Procedures: 


> Provide  explicit  and  public  statements 
about  institutional  expectations 

> Enable  the  institution  to  determine  the 
fit  between  expectations  and  the  level 
achieved 

> Provide  encouragement  and  the 
means  to  test  changes  that  could 
improve  services  & learning. 


Effective  Assessment  Methods  & Procedures: 


Provide  explicit  and  public  statements  about  institutional  expectations 

The  assessment  activities  should  be  based  on  explicit  institutional  commitments 
which  are  apparent  in  institutional  missions  and  policies  and  should  reflect 
institutional  expectations.  These  commitments  should  reflect  the  values  of  the 
institution  and  its  modes  of  operation. 


Enable  the  institution  to  determine  the  fit  between  expectations  and  the  level  achieved 

The  outcomes  of  the  assessment  should  help  the  institution  determine  if  the  service 
expectations  are  realistic,  if  the  work  processes  should  be  adjusted  to  achieve  a 
better  fit  between  expectations  and  results. 


Provide  encouragement  and  the  means  to  test  changes  that  could  improve  services  & 
learning. 

The  information  gleaned  from  the  assessment  should  show  directions  for 
change — it  should  enable  the  institution  to  prioritize  investment  and  to  change 
processes  for  the  better. 
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E-Metrics  Assessment  Plan 
Should 


> Have  administrative  clarity 

> Reflect  institutional  structure  and  staff 
functions 

> Balance  stakeholder  needs  with 
availability  of  data 

> Provide  for  the  input,  structure,  housing 
& archiving  of  data 

> Propose  a structure  to  disseminate  data, 
reports  & information — dynamic  Web 
intranet  (portal) 
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E-Metrics  Assessment  Plan  Should 


Have  administrative  clarity 

There  should  be  clarity  of  in  the  administrative  structure  for  assessment  There  should  be 
clear  lines  of  responsibility  and  accountability. 


Reflect  institutional  structure  and  staff  functions 

Assessment  plan  and  activities  should  reflect  the  institution’s  structure  and  the  functions 
of  the  staff  who  do  the  work.  Data  and  information  should  be  structured  to  reflect  the 
organization’s  structure. 


Balance  stakeholder  needs  with  availability  of  data 

Know  stakeholders  needs,  but  adjust  to  what  is  possible.  You  can  only  measure  based  on 
availability  of  data.  This  should  be  always  made  clear. 

Provide  for  the  input,  structure,  housing  & archiving  of  data 

Plan  to  organize  your  data  based  on  your  organization’s  needs  and  structure,  so  that  it 
makes  sense  to  those  who  use  it.  Plan  for  security  of  the  input  data,  for  archiving  the  data 
for  long  term  analysis. 


Propose  a structure  to  disseminate  data,  reports  & information  - dynamic  Web  intranet 
(portal) 

Involve  staff  in  developing  the  format  they  need  the  information  in.  It  is  important  that 
the  outcome  is  usable,  that  staff  who  need  the  information  receive  it,  search  it,  have  the 
capability  to  manipulate  it.  Plan  to  make  all  information  available  in  a Web  environment, 
possibly  in  a portal  environment 
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E-Metrics  Implementation 
Process 


> Preparation 

• Immediate 

• Long  term 

> Identification  of  Tasks,  Data,  and  Needs 

> Data  Collection 

> Information  Management 

> Reporting,  Dissemination  & Feedback 
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E-Metrics  Implementation  Process 


Preparation 

Immediate — short  term 
Long  term 

Prepare  the  institution  for  the  E-Metric  project. 


Identification  of  Tasks,  Data,  and  Needs 

Identify  all  the  activities,  stakeholder  need^  and  data  elements  of  the  project. 


Data  Collection 

Describe  in  detail  the  processes  of  data  collection. 


Information  Management 

Set  up  an  infrastructure  to  manage  information  and  to  analyse  information. 


Reporting,  Dissemination  & Feedback 

Set  up  an  infrastructure  to  create  and  use  reports.  Set  up  a mechanism  to  record  and 
analyse  feedback.  Set  up  a method  to  analyse  the  whole  process.  Close  the  feedback  loop. 
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Preparation 

Initial 


> Identify  Purpose  & Outcomes 

> Institutional  Level 

> Library  Level 

> Choose  Leaders  and  Provide  Training 

> Identify  Staff,  Train  and  Allocate 
Responsibilities 

> Identify  and  Train  Administrative  Support 
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Preparation:  Initial 
Identify  Purpose  & Outcomes 

Institutional  Level:  Mission,  vision,  general  goals  of  the  institution 
Libra  17  Level:  How  does  the  above  relate  and  inform  the  library’s  goals? 

Choose  Leaders  and  Provide  Training 

Identify  leaders  for  the  project  and  educate  them  about  its  purposes. 

Identify  Staff,  Train,  and  Allocate  Responsibilities 

Identify  units,  staff.  Allocate  responsibilities  and  provide  training. 


Identify  and  Train  Administrative  Support 

Identify  administrative  support  needed.  Define  responsibilities,  schedules,  training. 
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Preparation 

Detailing 


> Determine  Budget 

> Determine  IT  Hardware  & Software 

> Create  Plan  - Schedule 

> Specify  detailed  tasking 
>Set  long-term  goals 

> Set  short-term  objectives 
> Assign  responsibilities 


Preparation  - Detailing 


Determine  Budget 

Outline  initial  and  ongoing  budgets. 


Determine  IT  Hardware  & Software 

Identify  IT  needs.  Go  into  detail:  how  much,  where,  what  software,  support,  etc. 


Create  Plan  - Schedule 

Specify  detailed  tasking 
Set  long-term  goals 
Set  short-term  objectives 
Assign  responsibilities 


ERIC 
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Preparation 

Ongoing 


Build  a Culture  of  Assessment 

Assure  Leadership,  Planning,  Support 

Create  a Management  Information  System 

Invest  in  Staff  Development  & Training 

Develop  External  Links  - Multi-agency 

Reporting  System 

Network  Planning  - Integrate  into 

Institution’s  Plan 

Invest  in  (IT)  Infrastructure 

Merge  New  Network  Related  Measures  with 

Other  Assessment  Metrics 
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Preparation  - Ongoing 

This  should  be  done  all  the  time,  so  as  not  to  get  stale  and  forget  the  goals. 

Build  a Culture  of  Assessment 

Libraries  have  to  create  environments  that  enable  successful  assessment  and  the  implementation  of  results  based  on  these 
assessments.  In  other  words,  libraries  need  to  create  organizational  cultures  that  are  focused  on  effective  outcomes  for 
customers.  The  culture  change  needed  will  be  greatly  enhanced  by  the  adoption  of  a culture  of  assessment. 

How  do  we  create  aculture  of  assessment?  What  hinders  and  what  helps  us  in  this  endeavour?  Most  management  studies 
stress  the  importance  of  measurement  activities  for  ensuring  business  and  organizational  success. 

Assure  Leadership,  Planning,  Support 

Leadership  is  always  needed  in  order  to  get  a project  off  and  to  maintain  it  over  time.  Without  leadership,  this  process  of 
change  will  not  work.  Need  to  create  a framework  of  planning  and  support. 

Create  a Management  Information  System 

A Management  Information  Service  is  set  up  in  order  to  support  the  data  and  information  needs  of  the  organization.  Library 
leaders  who  possess  clearly  defined  expectations,  and  understand  the  need  for  data  and  information  to  support  decision 
making,  will  support  the  MIS.  The  MIS  unit  will  be  responsible  for  the  coordination  of  all  assessment  activities:  identification 
of  information  needs,  creation  of  an  appropriate  environment  for  organizing  data  and  information,  analysis  of  information  ,and 
making  information  available  to  the  processing  and  management  units  of  the  library. 

Invest  in  Staff  Development  & T raining 

The  changes  in  the  information  environment  point  to  the  need  for  continuous  upgrading  of  skill  sets  at  all  levels  of  the 
organization.  Without  a well- structured  learning  and  training  environment,  the  library  will  not  keep  up  with  new  information 
opportunities  and  will  be  less  effective.  Continuous  learning  is  becoming  part  of  the  job  of  each  person.  The  library  has  to 
plan  and  set  up  each  job  to  include  enough  time  and  opportunity  to  upgrade  skills.  Good  training  programs  also  boost  the 
confidence  level  of  staff  and  enable  effective  execution  of  change. 

Staff  development  in  all  areas  of  assessment  is  needed.  This  includes  training  on  appropriate  information  analysis  tools  and 
software  as  well  as  continuous  skills  upgrading  in  assessment  work.  Many  librarians  feel  that  they  lack  the  technical  and 
computer  skills  needed,  and  resist  acquiring  those  skills.  Many  feel  that  they  lack  the  skills  to  use  data  gathering  techniques 
such  as  surveys  and  focus  groups.  They  also  are  unsure  about  using  statistics.  Professionals  hesitate  working  in  areas  where 
they  lack  knowledge,  as  this  seems  to  signify  lack  of  control.  Lack  of  skills  also  creates  lack  of  confidence.  Much  assessment 
is  not  carried  out  because  staff  lacks  the  confidence  to  try  out  new  and  unfamiliar  activities.  The  skills  issue  may  also  be 
aggravated  by  a lack  of  well-oiganized  technical  support  for  non- technical  staff.  Without  this  supportive  environment,  staff’s 
capability  to  use  assessment  and  analysis  tools  is  derailed. 

Develop  External  Links  - Multi-agency  Reporting  System 

Creating  links  to  external  agencies  or  consortia  to  enable  consortia -based  reporting. 

Network  Planning  - Integrate  into  Institution’s  Plan 

Plan  to  integrate  the  library’s  network  into  the  parent  institution’s  plan  in  order  to  facilitate  creating  and  maintenance  of  data 
marts,  the  ability  to  link  easily  into  the  campus  student  system  and  financial  system.  This  will  also  facilitate  the  creation  of  an 
integrated  institutional  portal,  and  may  ease  the  analysis  of  learning  outcomes. 

Invest  in  (IT)  Infrastructure 

Look  at  IT  infrastructure  as  a long  term  investment  and  not  as  cost.  Invest  from  customer  service  perspective. 

Merge  New  Network  Related  Measures  with  Other  Assessment  Metrics 

Merge  the  E- Metrics  measures  with  other  traditional  metrics  to  see  more  clearly  service  trends. 
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Identification  of  Tasks,  Data, 
and  Needs 


> 

> 

> 

> 

> 

> 

> 


Identify  Stakeholders 

> External:  Local  Campus,  External  Agencies 

> Internal:  Staff,  Units  which  Need  information 
Schedule  Meetings  to  Identify  Data  Needs 
Identify  Report  Owners/Sponsors 
Identify  Sources  of  Data 

Identify  Inputs:  Create  Data  Dictionary 
Identify  Output  Format  and  File  Locations 
Identify  Support  Staff  (for  Data  Input,  etc.) 


Identification  of  Tasks,  Data,  and  Needs 
Identify  Stakeholders 

External:  Local  Campus,  External  Agencies 

Identify  campus  units  that  provide  or  receive  information,  as  well  as  external  agencies  (ARL). 
Internal:  Staff,  Units  which  Need  information 

Identify  functional,  process -based,  and  structural  units. 

Identify  individuals  in  each  area. 

Schedule  Meetings  to  Identify  Data  Needs 

Schedule  interview  with  each  unit  that  has  data  or  needs  information. 

Identify  Report  Owners/Sponsors 

Identify  in  each  unit  that  will  own  the  reports.  Who  will  get  the  reports  and  what  will  they  do  with  it? 

Identify  Sources  of  Data 

Identify  exactly  where  the  data  will  come  from  (external  vs.  internal  sources). 

Identify  Inputs:  Create  Data  Dictionary 

Create  a data  dictionary  for  type,  source,  format,  archive,  target  location,  etc. 

Identify  Output  Formatand  File  Locations 

Create  a list  of  outputs,  their  formatand  location. 

Identify  Support  StafT  (for  Data  Input,  etc.) 

Identify  staff  who’s  role  is  to  input  data  or  create  inputs  into  the  system.  Each  unit  or  functional  group  should 
identify  the  staff  with  explicit  responsibilities  for  data  input,  output,  etc. 
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Identification  of  Tasks,  Data, 
and  Needs 

Questions  for  Internal  & External  Stakeholders 


> What  Data  do  You  Need?  What  Level  of  Detail  is 
Needed?  What  Kind  of  Summary  Information  is  Needed? 

> Who  Will  Need  & Use  the  Data  and  Information? 

> How  Will  the  Data  be  Used?  For  What  Purpose? 

> When?  Schedule  of  Reports? 

> How  do  You  Want  to  Access  the  Information?  (Web, 
Spreadsheet,  Database?) 

> Identify  Need  for  Comparable  ARL,  IPED  & Other 
External  Data? 

> Timeline  for  E-Metrics  Project  Implementation? 
Schedule/  Comments? 


«• : ^ 
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Data  Collection 


> How/What  to  Collect  - Sources  of  Data 
> Systems  Based  - OPAC,  Web,  Vendor  Data 
(Canned  Reports,  Other  Reports) 

> Electron ic/Web  Resource  (costs  & use) 

> Non-Systems  (manual)  Library  Counts 
o Reference  Activities 
o ILL  Transactions 
o Document  Delivery 
o Turnstile  Counts 
o Other  Manual  Counts 

> Qualitative  Data  Sources  (Surveys,  Focus  Groups, 
Process  Analysis) 


11 


296 


Data  Examples 


Source:  TUG  (Tri-University  Group  of  Libraries  - University  of  Guelph,  Wilfrid  Laurier  University  & 
University  of  Waterloo)  SilverPlatter  ERL  (Electronic  Reference  Library)  Usage  Statistics 
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Network  Databases  1 JM/2000  - TUO  CSAUia 


Data  Examples 

Example  - Consortia  CSA  Database  Use  Data 
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Source:  TUG  (Tri-University  Group  of  Libraries  - University  of  Guelph,  Wilfrid  Laurler  University  & 
University  of  Waterloo)  CSA  (Cambridge  Scientific  Abstracts)  Usage  Statistics 
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Information  Management 


> Need  to  Create  a System/Structure  for  the  Long 
Term  Assessment 

> Need  a System/Structure  that  Enhances 
Decision  Making  & Planning 

> Need  a Management  Information  System 

> To  be  Led  by  Librarians!!! 
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Information  Management 

In  order  for  the  E-Metrics  system  to  work  long  term,  it  is  important  to  build  a structured 
management  system. 


Need  to  Create  a System/Structure  for  the  Long  Term  Assessment 

The  system  should  be  designed  for  the  long  term.  This  is  not  a short  term  project,  but  a system  that 
needs  to  be  ongoing.  It  needs  to  be  viewed  as  a necessity  for  the  library.lt  is  needed  for  an 
environment  that  supports  ongoing  assessment 


Need  a System/Structure  that  Enhances  Decision  Making  & Planning 

Systemic  means  that  it  is  strategic  and  it  is  necessary  for  a planning  environment  and  for  ongoing 
decision  making  at  all  levels  of  the  institution. 


To  be  Led  by  Librarians!!! 

This  is  not  a system  to  be  led  or  managed  by  systems  staff,  but  by  professional  librarians.  It  should 
be  led  by  librarians  who  care  about  customers,  about  analysis  of  data,  about  understanding  the 
links  and  the  relationships  they  find.  Only  librarians  would  do  it  justice. 


299 


Information  Management 

MIS/DSS  Systems 


> Provides  managers  and  staff  with  internal  and 
external  data  and  tools  for  the  analysis  of  the 
data. 

> Assist  in  the  decision  process 

> Act  as  support  tool  - does  not  replace 
managerial  judgement 

> Improve  effectiveness  of  decisions  is  the  main 
objective 
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MIS/DSS  Systems 

Management  Information  Systems  are  computer-based  tools  designed  to  improve 
management  decisions.  They  have  been  around  in  industry  and  business  since  the  1950s. 
Concurrent  with  technological  and  management  advances,  refinements  were  added,  and  these 
systems  evolved,  some  to  be  called  Decision  Support  Systems  (DSS)  and  later  Executive 
Information  Systems  (EIS).  The  need  for  systematic  application  of  Management  Information 
Systems  in  libraries  was  recognized  many  years  ago  by  Charles  McClure  and  others  (see 
attached  select  MIS  bibliography).  However,  systematic  application  of  some  kind  of  MIS 
in  the  library  environment  has  been  and  remains  rare. 


What  are  Management  Information  Systems?  What  do  they  do? 

•An  MIS  provides  managers  and  staff  with  internal  and  external  data  and  tools  for  the  analysis  of 
the  data.  These  are  software-based  systems  or  environments  which  use  input  measures,  and  with  the 
right  analysis  tools,  provide  information  to  managers. 

•They  assist  in  the  decision  process. 

•They  act  as  support  tools.  They  do  not  replace  managerial  judgement,  they  do  not  replace  humans. 
They  are  helping  tools. 

•Their  main  objective  is  to  improve  the  effectiveness  of  decisions.  In  other  words,  they  are  process 
improvement  tools,  information  tools  that  help  improve  outcomes. 
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Information  Management 

Objectives  (and  benefits)  of  a MIS 

> Assist  library  managers  and  staff  in  making 
better  decisions 

> Maintain  better  accountability  and  control  over 
library  resources 

> Monitor  and  control  resource  allocations 

> Improve  overall  library  effectiveness 

> Generate  internal  and  external  reports  as 
required 

> Improve  long  term  planning 

> Facilitate  performance  measures  activities 


Objectives  (and  Benefits)  of  an  MIS. 

In  the  library  setting,  what  could  be  the  objectives  and  eventual  benefits  of  an  MIS?  A short  list  could 
be  the  following: 

•First  assist  managers  and  staff  in  their  daily  decision-making  process.  With  better  information, 
decisions  will  be  more  effective. 

•Maintain  better  accountability  and  control  on  resources.  An  MIS  should  give  better  real-time 
information  for  resource  management,  and  allow  for  more  assured  accountability  of  the  library. 

•Monitor  and  control  resource  allocations  An  equitable  allocation  of  budgets,  especially  material 
budget  allocations,  is  always  a difficult  exercise.  A well-organized  MIS  may  improve  the  process, 
both  for  reliable  analysis  and  for  the  approval  process. 

•Improve  overall  library  effectiveness.  Paying  attention  to  outcomes,  to  improved  quality  of  work 
and  service  is  important.  Better  analysis  of  client  needs,  better  allocations  of  resources  and  services, 
better  management  decisions  should  improve  effectiveness  over  time. 

•Generate  internal  and  external  reports.  We  are  required  to  produce  lai^e  number  of  reports,  both 
internally  and  to  external  agencies.  Libraries  invest  large  staff  resources  in  collecting,  organizing, 
filing,  sending,  and  receiving  these  reports.  An  MIS  should  automate  most  of  these  activities  and 
execute  them  efficiently.  Hopefully  these  tools  will  also  enhance  comparison  analysis  between 
libraries. 

•Improve  Long  Term  Planning.  An  MIS  should  be  incorporated  into  the  planning  process,  or  be  the 
catalyst  to  enable  planning  where  it  is  not  present  or  not  yet  institutionalized. 

•Facilitate  Performance  Measures  Activities.  Help  with  other  qualitative  performance  measure 
activities  by  providing  reliable  information  and  tools. 


3 0 1 
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The  MIS  Information  Environment 


Leadership  & Staff  Resources 


HBM 


I — 

♦Data  Sources 


: non-systems  data  V 

i ♦Electronic  Resources 
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y\  ~£i*  web/  • 
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♦Processes  & Tools 


♦Outpu 


ts  :‘f 


♦Spreadsheets,  databases 
♦Business  Intelligence  Tools 
♦SQL  Queries 
♦Managed  Queries  Tools 
♦OLAP  Tools 

♦Statistical  Analysis  Tools 
♦Web  Analysts  Tools 


♦ Current  Data 
Repository 

♦ Executive  Information 

♦ Data  archive 

♦ Reports 

♦ Analysis 
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Outcomes  & Activities 


T 


♦Stakeholders  & Customers 


The  MIS  Environment 


Data  Sources  - Input  - from  where  we  derive  the  data 

Processes  & Tools  - how  we  manage  our  data  & information  and  the  tools  we  use  to  organize  and  analyze 
the  information. 


Outputs  - How  we  manage,  archive,  create  reports,  do  analysis  - create  an  analysis  and  decision-making 
loop. 

Feedback  Loop  — from  our  stakeholders  and  customers  to  library  leaders  and  staff. 
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Reporting  and  Dissemination 


> For  Whom  - Identify  Who  Will  Receive  Data 

> By  type  of  Information 

> By  Unit  Needs 

> How  - Identify  Tools  & Formats 

> Spreadsheets  (e.g.  Excel) 

> Databases  (e.g.  Access,  Impromptu,  PowerPlay) 

> Web  Tools  (e.g.  WebTrends) 

> Where  - Location  of  Systems  Reports,  Data  Files 

> Create  Data  Sets  for  Longitudinal  Analysis 
>Save  & Archive  Data  & Information 

>Make  Data  & Information  Available  on  Dynamic 
Web  Intranet 

> Review  & Adjust 
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Examples  of  Information  Analysis 

ERL  Databases  User  Logins  Analysis  in  a Consortia  Environment 
ERL  User  Logins 
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to  000 
80  000 
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20.000 

10.000 


— Totat UO  UW  Qct\*t 
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Source:  TUG  (Tri -University  Group  of  Libraries  - University  of  Guelph,  Wilfrid  Laurler  University  & 
University  of  Waterloo)  SllverPlatter  ERL  (Electronic  Reference  Library)  Usage  Statistics 
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Example  of  ERL  logins  by  TU  G libraries  over  a specific  time  period. 


Useful  to  analyze  overall  usage  as  well  as  to  compare  use  by  institutions. 


Gives  peak  and  low  time  picture. 
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Examples  of  Information  Analysis 
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Examples  of  Information  Analysis 

demographic  analysis  using  “local”  count  use 


School  or  Center  - 


The  20  more* 
used  bases 


The  20  less-  I 
used  bases  ' 


Source:  Univ.  of 
Pennsylvania  Libraries 
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Examples  of  Information  Analysis 

demographic  analysis  using  “local”  count  use 


Datab  ase  School  or  Center 
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Examples  of  Information  Analysis 


Analysis  of  use  by  hour  of  day 


- In-Library Stud*rn(-bu»»»  School* Cnmpu*  Mod  *m  Pool 


Source:  Univ.  of 
Pennsylvania  Libraries 
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Source:  Univ.  of 
Pennsylvania  Libraries 


Examples  of  Information  Analysis 

Analysis  of  use  by  domains  within  each  hour. 


Network  Domains: 

11 1 '■  In-Library 


►Studmt  House 


• Campus  Mod  an  Pool 


- All  Oha-  ( non- Penn  domain^ 


Thepointson  thisfanetotal  1 00%  of  logons  for  thohour5-6AM  .Inthis 

wampla  ’AllClha  Domain^  acnountoJ  (or  466%  of  attenpted  log-ons 
during  thegivei  hour. 
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The  E-Metrics  Assessment  Implementation  Loop 


Close  the  Assessment  Loop.  Every  implementation  is  a cycle  that  should  come  lull  circle.  What  you  learn 
should  be  reapplied  and  the  processes  re-examined  in  view  of  the  changes  found. 
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Additional  Potential 
Measures  & Issues 


> External  Vendor  Issues 

> Cost  Measures 

> Quality  Measures 

> Outcome  Oriented  Measures 


Additional  Potential  - Measures  & Issues 
External  Vendor  Issues 

How  to  gather  and  compare  information  from  many  vendors 

How  to  organize  information  from  many  vendors  into  one  coherent  report 

How  to  control  standards  of  reporting  by  external  vendors  or  timing  of  reports 

Cost  Measures 

How  to  understand  costs  over  time 
How  to  develop  unit  costs  by  local  client 

Quality  Measures 

How  to  identify  value  and  give  it  a quantifiable  measure  understood  by  funders 
Identify  quality  and  develop  assessment  methodologies  that  are  clear  and  understandable 
Measures  of  reliability  of  service — breakdown  of  workstations,  down  time  of  system 
Satisfaction  measures — qualitative 
Track  online  feedback  over  time 

Outcome  Oriented  Measures 

Measures  of  e-metrics  should  be  decoupled  from  traditional  library  measures 
We  should  be  able  to  measure  time  saved  by  use  of  electronic  resources 
Unit  cost  of  use  over  time 

Possible  better  research  outcomes  by  faculty  and  students 

Development  of  methodologies  to  measure  learning  outcomes  as  related  to  availability  of 
networked  services 
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External  Vendors  Data  & Reports 


> Diversity  of  Vendors 

> E-Journal  Publishers/Providers 

> E-Journal  Article  Databanks 

> E-Reference  Content  Providers 

> E-Book  Publishers/Providers 

> Some  combination  of  the  above 
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External  Vendor  Data 
Information  Issues 


> Consistency:  lack  of  standard  data 
definitions. 

> Report  formats:  requires  significant  efforts 
to  process  raw  data. 

> Frequency:  most  prefer  monthly  report. 

> Reliability  :some  vendors  are  better  than 
others.  There  is  some  indication  that 
vendors  are  looking  to  the  ICOLC  guidelines. 

> Changing  Electronic  Vendor  Statistical 
Reporting  Capabilities  (see  Appendix  D - 
ARL  E-Metrics  Project  Phase  One  Report) 
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External  Vendor  Data  Information  Issues 


Consistency  - lack  of  standard  data  definitions. 

Need  for  vendors  to  use  standard  data  definitions  in  order  for  libraries  to  be  able  to  create  macro 
reports  and  be  able  to  compare  between  use  of  resources  and  services  from  different  vendors. 


Report  formats  : requires  significant  efforts  to  process  raw  data. 

It  would  be  useful  to  develop  standard  reporting  formats  and  easy  ways  to  parse  data  into  local 
resources  for  further  analysis. 
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Additional  Measures  Costs 

> Hardware  & Software 

> new  & upgrades 

> public  & staff 

> Resource  Sharing:  associated  costs 

> Staff:  salaries,  training,  development, 
travel 

> Activity  Based  Costing 

> Special  Projects  (portals) 
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Additional  Measures  - Costs 


Hardware  & Software 


new  & upgrades 

public  & staff  (ARL  already  counts?) 


Resource  Sharing  - associated  costs 

Costs  of  ILL,  Document  Delivery,  Staffing,  Infrastructure,  consortia  fees,  special  arrangements.. 


Staff-  salaries,  training,  development,  travel 


Activity  Based  Costing 

It  may  be  useful  to  invest  in  ABC  in  order  to  get  more  realistic  picture  of  costs,  especially  unit 
costs  by  function,  service  costs,  and  develop  better  ratios  between  resources  and  activities.  There  is 
a learning  curve  as  not  many  institutions  of  higher  education  use  ABC. 


Special  Projects  (Portals) 

Developing  library,  enterprise,  or  knowledge  portals  is  expensive,  not  only  from  a staffing 
perspective,  but  also  from  a IT  resource  perspective.  If,  however,  we  see  it  as  a long  term 
investment  in  our  services  to  our  stakeholder  and  customers,  if  we  see  portals  as  a strategic 
necessity,  we  cannot  avoid  it.  It  is  doable  and  it  will  change  our  life. 
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Additional  Measures  Quality 


> Reliability 

> Staff  & Public  Workstations:  hours  of 
operation,  failure  rate,  no.  of  cold  re-boots 

> Public  Printing:  failure  rate 

> Count  of  system  related  problems  per 
day  by  problem  type 

> Analysis  of  customer  feedback 

> Analysis  of  proxy  server  problems 

> Measure  Quality  of  Existing  Resources 


Additional  Measures  - Quality 
Reliability 

Staff  & Public  Workstations:  hours  of  operation,  failure  rate,  no.  of  cold  re-boots 

Public  Printing:  failure  rate 

Count  of  system  related  problems  per  day  by  problem  type 

To  solve  reliability  of  systems  to  customers,  we  need  to  make  an  extra  effort  to  identify  what 
customers  want,  what  level  of  service  they  need,  and  where  we  fail. 

Analysis  of  customer  feedback 

Have  to  analyze  on  an  ongoing  basis  customer  feedback  and  not  see  it  as  a nuisance,  but  an 
opportunity  to  identify  problems,  see  what  it  takes  to  solve  them  and  respond  fast. 

Analysis  of  proxy  server  problems 

As  many  of  our  service  problems  relate  to  the  proxy  setups,  find  solutions  that  deal  with  these 
issues. 


Measure  Quality  of  Existing  Resources 

Assess  quality  of  e-metrics  resources  by  using  customer  feedback  and  benchmarking  the  services 
for  service  quality  measures. 
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Additional  Measures  We  Need 


> Outcomes:  Impact  of  Library  Network 
Services  on  the  Academic  Outcomes 

> Develop  Measures  to  Assess  Value  of 
the  Library  Gateway  to  “Public  Web” 

> Use  & Cost  Benefit  Analysis  of  Web 
Based  Services  (compare  to  traditional 
services) 

> Better  Understanding  of  Client 
Demographics  and  Needs 
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Critical  Needs  to  Support  Data  Collection, 
Analysis  & Reporting 


> Leadership:  Articulated  Purpose  & Support 

> Think  Systems,  Coherent  Outcomes  & Long 
Term  Commitment 

> Create  Good  Infrastructure 

> Understand  the  Need  for  a Management 
Information  Infrastructure 

> Populate  with  Competent  Skilled  & Professional 
Personnel 

> Assign  Systems  Support 

> Make  it  Happen!! 
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General  Conclusions 


> Executive  Support:  Continuous,  System 
Thinking,  Tolerance  to  Detail  & Ambiguity 

> Need  to  Educate  Staff  on  Culture  of 
Assessment:  Understanding  its  Purpose, 
Consensus  Building,  Common  Goals 

> Be  Aware  of  Local  vs.  Vendor  Perspectives  & 
Interests 

> Flexibility:  Allow  Risk  Taking  & Innovation 

> Involve  Stakeholders  & Customers:  Feedback 

> Create  Value:  ACT  on  What  You  Measurel 
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General  Conclusions 

Executive  Support:  Continuous,  System  Thinking,  Tolerance  to  Detail  & Ambiguity 

The  leadership  has  to  be  inclusive,  support  learning,  and  create  openness  between  people.  A 
leadership  steeped  in  culture  of  control  will  usually  deliver  either  no  change  or  will  reinforce  the 
status  quo  and  the  present  power  structure. 

The  leadership  understands  the  purpose  of  creating  a learning  organization,  it  understands  it  and  it 
believes  in  it  and  articulates  it  constantly  to  its  staff,  to  organizational  stakeholders,  and  to  external 
customers.  Needs  to  support  openness  and  trust  in  the  organization. 

Need  to  Educate  Staff  on  Culture  of  Assessment:  Understanding  its  Purpose,  Consensus  Building, 
Common  Goals 

Staff  have  to  be  empowered  for  real  change  to  take  hold.  Staff  need  a personal  stake  in  the  change. 
Only  then  will  they  embrace  change.  Change  is  resisted  because  it  is  “not  familiar”  and  is  viewed 
as  imposed  externally.  In  order  for  change  to  take  root,  leadership  has  to  create  organizational 
ownership. 

Culture  of  assessment  is  an  environment,  a process  that  encourages  learning.  Ownership  happens 
by  doing  and  seeing  results.  As  people  learn  how  to  do  assessment,  their  confidence  increases, 
they  see  results,  they  realize  that  they  are  the  creators  of  the  new  environment,  and  eventually  it 
spreads  through  the  organization. 

Be  Aware  of  Local  vs.  Vendor  Perspectives  & Interests 

Leaders  and  staff  have  to  be  cognizant  of  their  articulated  purpose  and  be  educated  about  vendors 
and  other  publishers  perspectives.  This  means  that  librarians  have  to  continue  to  dialogue  and  be 
well  informed  about  externals  pressures  in  order  to  make  good  decisions. 

Flexibility:  Allow  Risk  Taking  & Innovation 

Since  assessment  work  may  be  new  and  unfamiliar,  and  viewed  as  risky,  risk  should  be 
encouraged.  In  essence  risk  taking  will  have  to  be  the  norm  in  libraries.  Without  taking  chances 
and  experimenting  with  new  ideas  and  services,  we  will  fail.  Staff  should  be  encouraged  to 
experiment  and  possibly  fail.  You  learn  much  from  mistakes. 

Involve  Stakeholders  & Customers:  Feedback 

Continuous  communication  with  customers  is  maintained  through  needs  assessment,  quality 
outcome,  and  satisfaction  measurements.  Relevant  data  and  user  feedback  is  routinely  collected, 
analyzed,  and  used  to  set  priorities,  allocate  resources,  and  make  decisions.  Create  a learning 
environment  that  understands  the  need  for  the  feedback  loop  with  customers  and  appreciate  the 
need  for  assessment  as  an  everyday,  reflective,  systematic  activity. 


Create  Value:  ACT  on  What  You  Measure! 
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Clarifying  Expectations  for  Library 
Network  Assessment 

> Articulate  Expected  Outcomes 

> Identify  Where  Expected  Outcomes 
are  Addressed 

> Identify  & Collect  Baseline  Information 

> Determine  Methods  & Criteria  by 
which  the  Outcomes  will  be  Assessed 

> Articulate  Institution’s  Level(s)  of 
Expectation 


319 


34 


Preparation  for  Assessing 
Network  Services  is  as 
Important  as  the 
Assessment  Itself !!! 


ARL  New  Measures  Initiative 
The  E-Metrics  Project 

Recommended  Statistics  and 
Measures  for  Library 
Networked  Services 

ARL  E-Metrics  Phase  II  Report 
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Vendor  Statistics 

Working  Group 


> Invited  12  major  ARL  vendors  to  meet  in 
Denver  preceding  2000  ACRL  Meeting 


Academic  Press/IDEAL  * netLibrary 
Elsevier/Science  Direct  Silver  Platter  * 


Lexis/Nexis 

Ovid 

Bell  & Howell 
Gale  Group 
ISI  *t 


EBSCO 

JSTOR 

OCLC/First  Search 


* Unable  to  attend  Denver  Meeting 
t Nonparticipant  in  project. 
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A Framework  for  Developing  & Selecting 
Network  Statistics  & Measures 


> The  Network  Components 

> Technical  Infrastructure 

> Information  Content 

> Information  Services 

> Support 

> Management 
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The  Network  Component  Perspective  serves  as  a useful  mechanism  through  which  to  select  and  measure 
specific  aspects  of  a library's  networked  services  and  resources  along  a number  of  network  aspects. 


This  perspective  suggests  that  there  are  a number  of  measurable  components  to  electronic  networks: 

•Technical  infrastructure:  The  hardware,  software,  equipment,  communication  lines,  and  technical  aspects 
of  the  network  (e.g.,  workstations,  modems,  servers); 

•Information  content:  The  information  resources  available  on  the  network  (e.g.,  local  government 
information,  special  collections,  JSTOR,  ScienceDirect); 

•Information  services:  The  activities  in  which  users  can  engage  and  the  services  that  users  may  use  to 
complete  various  tasks  (e.g.,  online  reference  services,  usage  of  digital  information  content); 

•Support  The  assistance  and  support  services  provided  to  help  users  better  use  the  network  (e.g.,  training, 
help  desk); 

•Management  The  human  resources,  governance,  planning,  and  fiscal  aspects  of  the  network  (e.g., 
network  staff,  advisory  boards,  budgeting). 

These  network  components  provide  a means  through  which  to  consider  the  type  of  statistics  and  performance 
measures  that  would  enable  research  libraries  to  describe  and  evaluate  their  networked  services  and 
resources. 

It  is  possible  to  look  at  the  information  content  aspect  of  a network  and  consider  network  statistics  and 
performance  measures  for  that  aspect  of  library  networked  services,  thus  enabling  one  to  map  the  network 
statistics  and  performance  measures  as  presented  in  Part  4 of  this  report 
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The  notations  in  parentheses  here  and  in  this  slide  and  in  Figure  1.1  in  the  text  of  the  Phase  II  report  refer  to 
the  statistics  and  measures  presented  in  detail  in  Part  4 of  this  report. 

•Technical  Infrastructure 

•(D3)  Cost  of  digital  collection  construction  and  management; 

•Information  Content 

•(Dl)  Size  of  digital  library  collection; 

•(PI)  Percentage  of  electronic  reference  of  total  reference; 

•(P2)  Percentage  of  virtual  library  visits  of  all  library  visits; 

•(P3)  Percentage  of  electronic  books  to  all  books; 

•(Rl)  Number  of  electronic  full-text  journals; 

•(R2)  Number  of  electronic  reference  sources; 

•(R3)  Number  of  electronic  books; 

•(U5)  Number  of  virtual  visits  to  the  networked  library  resources; 


•Support 


> (D3)  Cost  of  digital  collection  construction  and  management; 


•Management 

•(Cl)  Cost  of  electronic  full-text  journals; 

•(C2)  Cost  of  electronic  reference  sources; 

•(C3)  Cost  of  electronic  books; 

•(C4)  Library  expenditures  for  bibliographic  utilities,  networks,  and  consortia; 
•(C5)  External  expenditures  for  bibliographic  utilities,  networks,  and  consortia; 
•(D3)  Cost  of  digital  collection  construction  and  management. 


Using  this  approach,  libraries  can  gain  a sense  of  the  use  and  uses,  management,  and  reach  of  their 
networked  services  and  resources  in  specific  areas  or  across  a number  of  areas.  In  addition,  libraries  can 
look  in-depth  at  particular  aspects  of  their  networked  services  and  resources.  Moreover,  by  selecting 
statistics  and  performance  measures  relevant  and  of  interest  to  library  staff  and  managers  required  for 
reporting  purposes,  or  other  motivational  factors,  it  is  possible  for  libraries  to  develop  an  overall  sense  of 
their  networked  services  and  resources  along  a network  dimension.  The  Network  Component  Perspective 
also  serves  as  a framework  for  the  development  of  additional  network  statistics  and  measures  not  field-tested 
during  this  project 


o 

ERIC 


325 


5 


Network  Component 
Framework 


Network 

Component 

ft,  f fc-A 

■]<?  < -4 . * v •; 

. .h  T •*  ‘ Network  Evaluation  Criteria  V./#  . 

• ! •: 1 

Extensiveness 

Efficiency 

Effectiveness 

Service 

Quality 

Impact 

Usefulness 

Adoption 

Technical 
Infra  structure 

Information 

Content 

Information 

Services 

Support 

Management 

BHBHR 
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Conceptualization  of  these  components  will  depend  on  the  particular  circumstances  of  the  individual  library. 
The  Network  Component  Model  (Bertot  and  McClure,  1998)  serves  as  a framework  for  such  a 
conceptualization  (see  Figure  1.2): 

• Extensiveness : How  much  service  the  library  provides  (e.g.,  number  of  users  accessing 
a Web  page  per  week,  number  of  database  sessions); 

• Efficiency : The  use  of  resources  in  providing  or  accessing  networked  information 
services  (e.g.,  cost  per  session  in  providing  access  to  remote  users  of  an  online  database, 
average  number  of  times  users  are  unable  to  successfully  connect  to  the  library’s 
servers); 

• Effectiveness : How  well  the  networked  information  service  meets  the  objectives  of  the 
provider  or  the  user  (e.g.,  success  rate  of  identifying  and  accessing  the  information 
needed  by  the  user); 

• Service  quality : How  well  a service  or  activity  is  done  (e.g.,  percentage  of  transactions 
in  which  users  acquire  the  information  they  need); 

• Impact  How  a service  made  a difference  in  some  other  activity  or  situation  (e.g.,  the 
degree  to  which  network  users  enhanced  their  ability  to  gain  employment  or  pursue 
business); 

• Usefulness : The  degree  to  which  the  services  are  useful  or  appropriate  for  individual 
users  (e.g.,  percentage  of  services  of  interest  to  different  types  of  user  audiences); 

• Adoption : The  extent  to  which  institutions  or  users  integrate  and  adopt  electronic 
networked  resources  or  services  into  organizational  or  individual  activities  (e.g., 
answering  reference  questions,  generating  interlibrary  loan  requests,  use  of  digital 
collections). 


It  may  not  be  necessary  to  develop  measures  that  address  every  evaluation  criteria  for  every  network 
component.  Instead,  the  matrix  serves  as  an  organizational  tool  for  evaluating  possible  measures  that  an 
individual  library  will  need  to  assess  its  outcomes  of  interest.  These  types  of  criteria  provide  an  important 
roadmap  for  thinking  about  the  type  of  data  element  and  statistics  that  would  be  necessary  to  produce  such 
measures,  as  well  as  providing  a quality  measurement  framework  for  library  networked  services  and 
resources. 
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Recommended  Statistics  & 
Measures 


> Patron  Accessible  Electronic  Resources  (R1-3) 

> Use  of  Networked  Resources  & Services  (U1*5) 

> Expenditures  for  Networked  Resources  & Related 
Infrastructure  (Cl -5) 

> Library  Digitization  Activities  (D1-3) 

> Performance  Measures  (PI -3) 


This  section  provides  definitions,  data  collection  procedures,  and  discusses  related  issues  pertaining  to 
interpreting  and  using  the  recommended  statistics  and  measures.  The  definitions  and  procedures  were 
derived  from  a month  of  field-testing  at  more  than  a dozen  ARL  libraries.  The  statistics  and  performance 
measures  represent  a minimum  set  of  data  that  need  to  be  collected  continually  and  used.  Individual  libraries 
will  need  to  develop  local  procedures  to  support  data  collection  activities  within  the  guidelines  of  this 
manual.  However,  readers  need  to  recognize  that  the  statistics  and  measures  will  be  refined  and  extended 
continuously  in  the  future. 

4.1  Recommended  Statistics  and  Measures 

Table  4.1  Network  Statistics 

Patron  Accessible  Electronic  Resources 

R1  Number  of  electronic  full-text  journals 

R2  Number  of  electronic  reference  sources 

R3  Number  of  electronic  books 

Use  of  Networked  Resources  and  Services 

U 1 Number  of  electronic  reference  transactions 

U2  Number  of  logins  (sessions)  to  electronic  databases 

U3  Number  of  queries  (searches)  in  electronic  databases 

U4  Items  requested  in  electronic  databases 

U5  Virtual  visits  to  library’s  Web  site  and  catalog 

Expenditures  for  Networked  Resources  and  Related  Infrastructure 

C 1 Cost  of  electronic  full-text  journals 

C2  Cost  of  electronic  reference  sources 

C3  Cost  of  electronic  books 

C4  Library  expenditures  for  bibliographic  utilities,  networks,  and  consortia 

C5  External  expenditures  for  bibliographic  utilities,  networks,  and  consortia 

Library  Digitization  Activities 

D1  Size  of  library  digital  collection 

D2  Use  of  library  digital  collection 

D3  Cost  of  digital  collection  construction  and  management 

Table  4.2  Performance  Measures 

Performance  Measures 

P 1 Percentage  of  electronic  reference  transactions  of  total  reference 

P2  Percentage  of  virtual  library  visits  of  all  library  visits 

P3  Percentage  of  electronic  books  to  all  monographs 
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Patron  Accessible  Electronic 
Resources 


> R1  - Number  of  electronic  full-text 
journals 

> R2  - Number  of  electronic  reference 
sources 

> R3  - Number  of  electronic  books 


■■■  ■■  .(M  .y  y .....g, ^ ■ ■ V1  ' 


4.3.1  Statistics  Related  to  Patron  Accessible  Resources 

The  developed  statistics  for  patron  accessible  resources  account  for  networked  resources  and  services.  The 
current  ARL  membership  criteria  index  lacks  separate  measures  for  electronic  and  networked  monographs, 
serials,  and  bibliographic  utilities.  Though  these  electronic  and  networked  resources  may  limit  the  amount 
of  print  materials  acquired  and  may  cost  more  than  their  print  counterparts,  they  do  constitute  more  widely 
available  resources. 


In  the  electronic  and  networked  realm,  the  more  a library  has,  the  more  materials  are  provided  to  customers 
anytime  and  anywhere.  Although  local  needs  and  available  resource  allocations  may  differ  from  library  to 
library,  the  resource  statistics  allow  academic  research  libraries  to  see  and  to  demonstrate  to  others  the 
changing  nature  of  library  collections  over  the  years.  In  turn,  the  libraries  are  expected  to  use  them  to  make 
decisions  about  resource  allocations  (budget,  staff  time,  etc.)  and  to  undertake  strategic  planning 
accordingly.  Furthermore,  the  picture  of  available  resources  provides  libraries  with  an  opportunity  to  offer 
valued  services.  However,  because  the  evolving  nature  of  these  statistics  will  rely  heavily  on  technological 
enhancements,  all  libraries  are  encouraged  to  use  extra  caution  while  serving  their  institutional  goals, 
missions,  and  visions. 
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R1  - Number  of  Electronic  Full-text 
Journals 


y Definition  - Number  of  electronic  full-text 
journal  subscriptions  - by  individual  institution 
or  consortia  licensing. 


* /.  -•Cy'-C 


y Rationale  - Documents  degree  of 
expansion  of  electronic  subscriptions 
available.  Can  be  used  to  show  good 
coverage  & need  for  more  funding. 
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Definition:  Number  of  electronic  full-text  journal  subscriptions  that  the  library  provides  to  users  either 
through  an  individual  institutional  licensing  contract  with  the  provider  of  journals  or  through  other 
arrangements  (e.g.,  regional  or  state  consortium)  for  which  the  library  pays  a reduced  or  no  fee  for  access. 


The  full-text  journals  should  provide  both  search  and  browse  capabilities  by  title  and  issue.  This  is  different 
from  journal  article  databases,  such  as  Expanded  Academic  ASAP  in  INFOTRAC,  that  do  not  provide 
browsing  capability. 


This  includes  electronic  full-text  journals  offered  by  established  scholarly  journal  publishing  houses  (e.g., 
Elsevier’s  ScienceDirect  and  Academic  Press’s  IDEAL),  scholarly  societies  (e.g.,  American  Chemical 
Society  journals  and  American  Institute  of  Physics  Online),  and  services  which  aggregate  content  from 
smaller  publishers  or  from  those  publishers  that  prefer  to  use  an  external  delivery  platform  (Highwire,  OCLC 
ECO,  and  EbscoOnline).  This  should  exclude  general-purpose  periodicals  such  as  magazines  and 
newspapers. 


Rationale:  Electronic  access  has  expanded  dramatically  to  provide  a range  of  useful  resources  for  library 
users.  This  statistic  helps  document  the  degree  of  expansion  of  electronic  resource  availability  in  the 
individual  library  and  can  be  used  to  justify  continuation  and  enhancement  of  these  services. 


Research  libraries  act  increasingly  as  gateways  to  a vast  array  of  external  information.  This  measure 
specifically  addresses  the  extensiveness  of  scholarly  content  a library  provides  to  its  user  community.  In 
many  cases,  electronic  access  enables  the  library  to  offer  larger  selections  of  journals  than  it  could  provide  in 
paper  format.  This  statistic  can  also  be  used  for  library  promotion  and  internal  and  external  reporting. 
Particularly,  this  statistic  aims  at  showing  the  changing  nature  of  traditional  scholarly  resources  with 
improved  and  better  access  anytime  and  anywhere. 
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R1  - Number  of  Electronic  Full-text 
Journals 


y Data  source  - local  or  vendors. 

y*  Frequency  - annual,  monthly,  etc. 

^ Process  - parse  into  database  or 
spreadsheet,  update  dynamically  from  local 
catalog  or  vendor  record. 
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Implementation 


Collected  by:  Local  and  vendors 


Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly,  quarterly) 


Procedures:  It  is  impossible  to  obtain  the  complete  list  of  electronic  full-text  journals  from  a single  source. 
Possible  sources  for  the  information  include  library  catalog  records  (those  records  that  point  to  Web 
addresses),  library  Web  pages  that  list  the  journal  titles,  the  internal  electronic  resource  management 
database,  and  vendor  records  (Web  sites  and  contract  documentation). 

• Create  a master  list  of  full-text  electronic  journals  from  all  the  sources  available.  Use  a spreadsheet  or 
database  program  to  organize  and  maintain  the  list 

• Remove  titles  that  do  not  meet  the  above-mentioned  definition  but  keep  duplicate  titles  resulting  from 
multiple  subscriptions  (e.g.,  main  library  and  medical  or  law  libraries). 

• Record  the  counts  and  be  sure  they  are  updated  regularly.  The  library  should  also  update  the  count 
information  on  the  library  Web  site  and/or  in  marketing  brochures  on  a regular  basis. 


Special  considerations:  It  is  time-consuming  to  establish  procedures  to  collect  this  statistic  for  the  first  time. 
However,  once  that  is  done,  it  will  be  relatively  easy  to  update  the  information.  (This  applies  to  other 
statistics  and  measures  included  in  the  manual  as  well.) 


Include  journal  titles  that  come  with  print  subscriptions  or  print  plus  online  subscriptions  since  the  focus  of 
the  statistic  has  to  do  with  how  many  scholarly  electronic  journal  titles  users  can  access.  Do  not  include 
free  government  publications  and  free  electronic  journals  to  which  the  library  provides  links. 


Free  government  publications  and  free  electronic  journals  are  a valuable  resource  for  many  libraries.  How  to 
collect  statistics  relating  to  these  resources  will  be  addressed  in  the  future. 
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R2  - Number  of  Electronic  Reference 
Resources 


^ Definition  - number  of  electronic  reference 
resources  & aggregation  services  - by 
individual  institution  or  consortia  licensing. 

^ Rationale  - documents  degree  of  expansion 
of  electronic  resources  available.  Can  be 
used  to  show  good  coverage  & need  for  more 
funding. 


RMMNNMI  &-■ 


Definition  - This  includes  citation  indexes  and  abstracts;  full-text  reference  sources  (e.g.  encyclopedias, 
almanacs,  biographical  and  statistical  sources,  and  other  quick  fact-finding  sources);  full-text  journal  and 
periodical  article  collection  services  (e.g.,  EBSCOhost,  ProQuest,  Academic  Universe,  and  INFOTRAC 
OneFile);  dissertation  and  conference  proceedings  databases;  and  general-purpose  magazines  and 
newspapers.  Licensed  electronic  resources  also  include  those  databases  that  institutions  mount  locally. 


Rationale:  Networking  technology  in  libraries  has  improved  and  increased  dramatically  user  access  to  a 
range  of  useful  reference  resources.  This  statistic  documents  the  degree  of  expansion  of  electronic  resource 
availability  and  can  be  used  to  justify  continuation  and  enhancement  of  these  services.  In  the  1990s,  because 
of  the  increasing  popularity  of  the  Internet,  the  ways  reference  interviews  were  held  and  reference  sources 
were  used  changed  Today,  users  have  electronic  formats  as  well  as  traditional  reference  sources  to  provide 
answers  to  their  reference  questions. 


Research  libraries  traditionally  act  as  gateways  to  a vast  array  of  external  informatioa  This  measure  deals 
with  the  extensiveness  of  scholarly  content  the  library  provides  to  the  user  community  and  the  availability  of 
reference  sources  on  an  anytime/anywhere  basis.  In  many  cases,  electronic  access  enables  the  library  to 
offer  more  resources  than  it  could  in  paper  format  This  statistic  can  also  be  used  for  library  promotion  and 
internal  and  external  reporting.  Specifically,  this  statistic  aims  at  showing  the  changing  nature  of  traditional 
scholarly  resources  with  improved  access. 
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R2  - Number  of  Electronic  Reference 
Resources 


> Unit  of  Measure  - database  - not  the 
service. 

^ Data  source  - local  or  vendors. 

> Frequency  - annual,  monthly,  etc. 

y Process  - parse  into  database  or 
spreadsheet,  update  dynamically  from  local 
catalog  or  vendor  record. 


Implementation 

Collected  by:  Local  and  vendors 


Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly,  quarterly) 


Process  - As  in  the  case  of  the  number  of  electronic  full-text  journals,  it  is  impossible  to  obtain  the  complete 
list  of  databases  from  a single  source.  Possible  sources  for  the  information  include  library  catalog 
records  (those  records  that  point  to  Web  addresses),  library  Web  pages  that  list  the  database  titles,  the 
internal  electronic  resource  management  database,  and  vendor  records  (Web  sites  and  contract 
documentation). 


• Create  a master  list  of  electronic  databases  from  all  the  sources  available.  Use  a spreadsheet  or  database 
program  to  organize  and  maintain  the  list. 

• Remove  titles  that  do  not  meet  the  above-mentioned  definition  but  keep  duplicate  titles  resulting  from 
multiple  subscriptions  (e.g.,  main  library  and  medical  or  law  libraries). 

• Record  the  counts  and  be  sure  they  are  updated  regularly.  The  library  should  also  update  the  count 
information  on  the  library  Web  site  and/or  in  marketing  brochures  on  a regular  basis. 


Special  considerations:  The  unit  of  measurement  here  is  the  database  not  the  whole  service  provided  by  a 
vendor.  For  example,  if  the  library  subscribes  to  OVID  and  the  company  provides  five  databases 
(ABI/Inform,  Books  in  Print,  CINAHL,  INSPEC,  and  PsycINFO),  then  the  count  is  5,  not  1 . By  the 
same  token,  if  the  library  subscribes  to  three  database  packages  (Academic  Universe,  Congressional 
Universe,  and  Statistical  Universe)  from  Lexis-Nexis,  the  count  is  3. 


This  count  should  not  include  freely  available  databases  to  which  the  library  provides  links  or  library -created 
finding  aids. 


Freely  available  databases  and  library-created  finding  aids  are  a valuable  resource  for  many  libraries.  How 
to  collect  statistics  relating  to  these  resources  will  be  addressed  in  the  future. 
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R3  - Number  of  Electronic 
Books 

> Definition  - # of  full-text  monographs  - by 
individual  institution  or  consortia  licensing. 

> Rationale  - documents  degree  of 
expansion  of  e-books  - to  be  used  with 
caution  as  tech  & use  evolves. 


Definition:  Number  of  electronic  full-text  monographs  that  the  library  offers  to  its  users  either  through  an 
individual  licensing  contract  with  the  content  providers  or  through  other  arrangements  (e.g.,  regional  or  state 
consortium)  where  the  library  pays  a reduced  or  no  fee  for  access. 


This  includes  electronic  books  purchased  through  vendors,  such  as  netLibrary  and  Books24x7,  and 
electronic  books  that  come  as  part  of  aggregate  services.  It  excludes  internally  digitized  electronic  books, 
electronic  theses  and  dissertations,  digitally  created  archival  collections  (e.g.,  Early  English  Books  Online), 
and  other  special  collections.  This  also  excludes  publicly  available  electronic  books  to  which  the  library 
provides  Web  links.  It  does  not  include  machine-readable  books  distributed  on  CD-ROM,  or  accompanied 
by  print  books. 


Rationale:  Networking  technology  in  libraries  has  improved  and  increased  dramatically  user  access  to  the 
electronic  counterparts  of  some  traditional  sources.  This  statistic  documents  the  degree  of  expansion  of  e- 
books.  In  the  mid  90s,  networking  and  resource  sharing  technologies  provided  libraries  with  print  books  and 
e-books  that  were  made  available  through  a library’s  networks. 


Because  the  evolving  nature  of  this  statistic  will  heavily  depend  on  technological  enhancements,  all  libraries 
are  encouraged  to  use  extra  caution  while  pursuing  their  institutional  goals,  missions,  and  visions. 

Moreover,  the  definition  of  e-books  is  still  evolving.  This  statistic  is  an  early  attempt  to  keep  track  of  this 
type  of  resource  as  it  becomes  more  widely  available. 
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R3  - Number  of  Electronic 
Books 


> Unit  of  Measure  - title  count. 

> Data  source  - local  or  vendors. 

> Frequency  - annual,  monthly,  etc. 

> Process  - parse  title  into  database  or 
spreadsheet,  update  dynamically  from  local 
catalog  or  vendor  record.  Count  duplicate 
titles. 

> Related  Issues  - evolving  tech,  location, 
access,  use  vs.  circulation,  etc. 


Implementation 

Collected  by:  Local  and  vendors 

Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly,  quarterly) 

Procedures: 

• For  each  electronic  book  collection,  get  the  electronic  title  counts  from  either  the  providers  or  catalog 
records.  Unlike  electronic  full-text  journals  and  reference  databases,  it  is  not  necessary  to  list  the  titles 
for  each  electronic  book  collection. 

• Count  any  duplicate  titles  resulting  from  multiple  subscriptions  (e.g.,  main  library  and  medical  or  law 
libraries). 

• Record  the  counts  and  be  sure  they  are  updated  regularly.  The  library  should  also  update  the  count 
information  on  the  library  Web  site  and/or  in  marketing  brochures  on  a regular  basis. 

Special  considerations:  Do  not  include  book  collections  that  are  a part  of  aggregate  services  and  function 
more  as  a reference  collection  (e.g.,  MD  Consult  reference  books,  ProQuest’s  Early  English  Books 
Online,  and  books@OVID).  They  should  be  reported  in  the  electronic  reference  databases. 

Do  not  include  freely  available  electronic  books  such  as  titles  available  from  the  National  Academy  Press. 

Related  issues:  Electronic  books,  still  evolving  in  terms  of  technology  and  adoption  for  use,  present  a 

number  of  issues  in  terms  of  definition  and  measurement,  such  as  "location,"  accessibility  (metadata  and 
access  points),  and  use  versus  circulation  (e.g.,  is  online  use  for  20  minutes  a circulation,  as  it  would  be 
with  reserve  materials;  or  does  a circulation  of  electronic  books  require  a minimum  period  of  use,  such 
as  24  hours?). 

• What  about  reference  book  collections  provided  by  vendors?  Should  they  be  treated  as 
electronic  books,  for  example,  or  should  they  be  treated  as  a database,  on  the  grounds  that  they 
are  used  as  databases? 

• Count  only  those  books  that  a user  can  check  out,  as  they  would  traditional  books.  Unlike 
traditional  books  that  the  library  purchases  and  owns,  electronic  books  can  be  subscribed  to  for 
an  ongoing  fee.  In  this  case,  the  library  accounting  system  may  treat  these  as  serials  rather 
than  books  because  of  the  type  of  payment.  It  is  relatively  easy  to  keep  track  of  the  number  of 
electronic  books  right  now  since  most  libraries  deal  with  only  a handful  of  e-book  vendors, 
such  as  netLibrary  and  Books24x7.  But  in  the  future,  it  will  become  increasingly  difficult  to 
do  this  as  the  sources  of  electronic  books  proliferate. 


Finally,  some  provisions  of  contractual  agreements  between  libraries  and  vendors  may  limit  the  level  of  use 
of  e-books.  These  issues  need  be  addressed  in  future  research. 
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Use  of  Networked  Resources  & 
Related  Infrastructure 


> U1  - Number  of  electronic  reference 
transactions 

> U2  - Number  of  logins  (sessions)  to 
electronic  databases 


> U3  - Number  of  queries  (searches)  in 
electronic  databases 

> U4  - Items  requested  in  electronic 
databases 


> U5  - Virtual  visits  to  library’s  Web  site 
and  catalog 
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4.3 1 Statistics  Related  to  Use  of  Electronic  Networked  Resources  and  Services 

High  use  of  a library  resource  or  service  implies  a collection  development  program  that  is  working  to  create 
access  to  the  resources  customers  need.  Use  and  the  need  can  also  identify  resources  and  services  that  are 
seen  as  particularly  valuable  in  the  education  and  research  enterprise  and  should  be  expanded,  or  perhaps 
resources  and  services  that  should  be  discontinued  due  to  lack  of  use  and  interest.  Whether  provided  by 
vendors  or  collected  institutionally,  usage  statistics  can  help  a library  administrator  make  decisions  and  plan 
for  the  future  in  order  to  meet  not  only  users'  expectations  and  needs  but  also  institutional  goals.  The 
reported  data  can  also  provide  other  information  as  to  where  and  when  people  use  the  library’s  materials  and 
how  well  the  library  serves  its  target  audience  and  anticipates  their  potential  needs. 


The  cost  of  providing  access  to  networked  resources  and  services  can  be  more  expensive  than  that  of 
traditional  counterparts.  Depending  heavily  on  earlier  ICOLC  guidelines,  the  E-Metrics  use  measures  put 
this  in  the  perspective  of  the  changing  academic  research  library  environment.  The  purpose  of  the  use 
measures  is  to  provide  statistics  relating  to  the  use  of  networked  services  and  resources.  Therefore,  it  is 
expected  that  library  administrators  can  reconsider  some  resource  allocation  issues  as  the  number  of 
resources  and  services  tend  to  increase  while  people  are  provided  greater  access.  Please  note  that,  as  with 
most  of  the  statistics  in  this  study,  statistics  related  to  the  use  of  library  resources  and  services  should  be 
revisited  and  perhaps  modified  as  the  technology  advances. 
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U1  - Number  of  Electronic  Reference 
Transactions 


> Definition  - number  of  electronic  reference 
transactions  - via  e-mail,  WWW  form,  etc. 

> Rationale  - libraries  are  interested  in 
tracking  the  development  of  new  electronic 
services.  Attempt  to  measure  reference 
transactions  through  new  electronic  tools 
and  services. 


Definition:  Number  of  electronic  reference  transactions  conducted  via  email,  a library’s  Web  site,  or  other 
network  communications  mechanisms  designed  to  support  electronic  reference.  An  electronic  reference 
transaction  must  include  a question  either  received  electronically  (e.g.,  via  e-mail,  WWW  form,  etc.)  or 
responded  to  electronically.  Those  transactions  that  are  both  received  and  responded  to  electronically  are 
counted  as  one  transaction.  This  count  excludes  phone  and  fax  traffic  unless  either  the  question  or  answer 
transaction  occurs  via  the  described  manner.  It  includes  the  counts  accrued  from  participation  in  any  local 
and  national  projects,  such  as  DigiRef  and  the  Library  of  Congress's  CDRS  (Collaborative  Digital  Reference 
Service). 


A reference  transaction  is  an  information  contact,  which  involves  the  knowledge,  use,  recommendations, 
interpretation,  or  instruction  in  the  use  of  one  or  more  information  sources  by  a member  of  the  library  staff. 


Rationale:  Libraries  are  making  more  of  their  services  available  electronically  and  are  interested  in  tracking 
the  development  of  a new  and  emerging  library  service.  There  is  a need  to  better  document  this  transition  to 
facilitate  and  improve  resource  allocation  activities.  This  statistic  represents  reference  activities  conducted 
electronically  in  the  library.  It  is  an  attempt  to  measure  reference  transactions  through  new  electronic  tools. 
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OU1  - Number  of  Electronic  Reference 
Transactions 

> Unit  of  Measure  - request  count,  time  it  took. 

> Data  Source  - local  server,  manual  tally,  e-mail 
count. 

> Frequency  - daily,  monthly,  annually,  etc. 

> Process  - clarify  process,  identify  activity  points, 
identify  collectors  of  data,  consolidate  data. 

> Related  Issues  -This  measure  may  have  to  be 
broken  down  into  additional  data  types  - time, 
type  of  query,  type  of  interaction,  scheduling 
issues,  measures  of  quality  and  reliability. 


17 


Implementation 

Collected  by:  Local 

Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (monthly,  quarterly).  This 
statistic  can  be  collected  in  the  same  manner  as  the  library  gathers  other  reference  transactions  data. 

Procedures: 

• Select  a typical  week  (or  month)  to  run  a sample  study.  Be  sure  to  vary  the  specific  week  (or  month) 
chosen  over  the  course  of  a year  or  from  year  to  year  to  account  for  seasonal  fluctuations. 

• Key  tasks  include  distributing  a daily  tally  sheet,  collecting  the  daily  tally  sheet,  adding  each  day’s  totals 
to  a weekly  figure,  and  being  available  to  respond  to  data  collection  problems  should  they  occur. 

• Transactions  may  be  via  e-mail,  a form  on  a Web  page,  etc.  Electronic  reference  transactions  may 
involve  more  than  reference  desk  staff  (e.g.,  Web  master,  various  reference  personnel,  library  director, 
volunteers,  etc.).  Establish  an  administrative  procedure  to  report  electronic  reference  transaction  counts 
to  a designated  staff  person,  no  matter  who  receives  the  questions  or  answers  the  reference  requests. 

• Disseminate  the  new  procedure  and  rationale.  Several  notices  throughout  the  year  may  be  necessary. 

• Report  an  electronic  reference  transaction  as  you  would  a face-to-face  reference  transaction.  Thus,  one 
e-mail  request  may  contain  several  reference  questions  taking  varying  times  to  complete.  For  example, 
one  e-mail  request  could  contain  two  relatively  short  reference  questions  and  one  reference  question  that 
took  10-15  minutes  to  answer.  Count  the  number  of  requests,  not  the  number  of  questions.  Thus,  in  the 
example  you  would  report  one  (1)  as  the  number  of  electronic  reference  transactions  even  though  there 
were  three  questions.  Report  counts  using  pre-established  local  library  reporting  periods  (weekly, 
monthly,  etc.). 

• Indicate  and  describe  any  additional  methods  used  outside  of  this  definition  and  guidelines. 

Special  considerations:  Unless  the  library  uses  electronic  reference  management  software  to  collect  and 
report  transaction  data,  it  is  difficult  to  keep  track  of  a complete  reference  transaction  cycle  (query  and 
response)  because  of  time-delays  and  the  involvement  of  several  parties. 

As  stated  in  the  definition,  the  statistic  includes  the  number  of  service  transactions  provided  to  patrons 
outside  the  university  or  the  parent  institution  that  the  library  serves,  through  regional  or  national 
cooperative  efforts  and  through  library  policies. 

Related  issues:  Reference  services  are  undergoing  rapid  changes.  Libraries  are  experimenting  with  different 
modes  of  electronic  reference.  One  could  say  that  simple  email  transactions  that  are  prominently 
mentioned  in  the  procedures  are  not  much  different  from  traditional  reference  services.  How  can  a 
library  measure  quality  in  providing  different  types  of  electronic  reference  services  such  as  live-chat 
with  text/vo ice/ video?  Will  this  measure  help  the  library  determine  user  demand  and  thereby  plan  for 
resource  allocation?  To  answer  these  questions,  libraries  need  to  collect  more  detailed  information  such 
as  length  of  time  taken  to  answer  questions,  types  of  questions  by  types  of  transactions,  and  so  on.  Also, 
this  statistic  is  likely  to  produce  some  useful  figures  and  trends  regarding  staff  support  and  allocation  in 
reference  activities.  ’ } > ' ‘ 
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U2  - Number  of  Logins  (Sessions) 
to  Electronic  Databases 


> Definition  - number  of  user  initiated  sessions  in 
licensed  electronic  resources.  Starts  at  connection 
& ends  with  explicit  termination  (timeout  or  logout). 

> Rationale  - will  give  data  about  relative  use  of 
each  database.  Will  show  use  of  networked 
resource. 

> Related  Issues  - need  vendors  to  report  by 
agreed  standards.  Need  to  understand  why  counts 
fluctuate,  changes  in  the  database  itself,  type  of 
license,  infrastructure  stability. 


pff# 


Definition:  Number  of  user  initiated  sessions  in  licensed  electronic  resources.  A session  or  login  is  one 
cycle  of  user  activities  that  typically  starts  when  a user  connects  to  a database  and  ends  with  explicit 
termination  of  activities  (by  leaving  the  database  through  logout  or  exit)  or  implicit  termination  (time  out  due 
to  user  inactivity).  Licensed  electronic  resources  also  include  those  databases  that  institutions  mount  locally. 


Rationale:  One  purpose  of  having  a networked  environment  is  to  promote  connectedness  and  accessibility  to 
a variety  of  information  resources,  hence  the  need  for  this  measure.  Also,  the  gradual  shift  in  the  materials 
expenditures  from  traditional  print-based  resources  to  electronic  databases  can  be  understood  with  the 
measure.  This  measure  will  produce  a count  of  how  often  specific  databases  are  used  and  complement 
traditional  physical  attendance  counts. 


Related  issues:  When  analyzing  the  login  counts,  it  might  be  important  to  explain  any  increases  or  decreases 
in  the  figures.  Specify,  for  example,  whether  the  increase  comes  from  (1)  the  addition  of  new  databases,  (2) 
databases  which  did  not  report  the  statistic  in  the  past  but  have  now  begun  reporting,  (3)  increased  demand, 
and/or  (4)  an  increase  in  the  number  of  simultaneous  users. 


Problems  with  the  comparability  of  login  counts  from  different  vendors  is  a serious  threat  to  the  utility  of  the 
combined  count.  Content  providers  use  different  time-out  thresholds  (ranging  from  7 to  30  minutes  on 
average).  Also,  because  of  the  IP-based  authentication,  several  sessions  conducted  at  the  same  public 
workstation  can  be  counted  as  a single  login.  Alternatively,  libraries  can  collect  attempted  logins  to  various 
licensed  databases  by  making  users  go  through  a central  gateway  (which  counts  all  attempted  logins).  This 
will  ensure  that  one  login  attempt  to  a database  is  the  same  as  a login  to  other  databases.  However,  what  this 
data  collection  method  misses  is  user  logins  that  go  directly  to  content  provider  sites.  It  is  unclear  how  many 
user  logins  fell  into  this  category,  but  the  phenomenon  certainly  results  in  a substantial  undercount  of  user 
logins. 


While  the  gross  login  figure  is  useful,  it  is  useful  only  for  trend  plotting  and  gross  justification  of  electronic 
resources.  Within  the  library,  the  usage  measures  of  licensed  electronic  resources  have  many  users  and  uses, 
Circulation  of  usage  statistics  on  the  database  title  level  (or  in  an  extreme  case  on  the  journal  title  level)  and 
discussion  of  any  noticeable  changes  (or  lack  thereof)  need  to  occur  at  various  levels  among  the  concerned 
parties,  including  collection  development  personnel,  Web  master(s),  technical  services  staff,  and  so  on. 
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U2  - Number  of  Logins  (Sessions) 
to  Electronic  Databases 

> Unit  of  Measure  - Logins  or  session 
counts. 

> Data  Source  - vendor  report. 

> Frequency  - daily,  monthly,  annually,  etc. 

> Process  - get  monthly  usage  data  from 
vendors,  copy  or  parse  data  for  each 
database  to  an  in-house  spreadsheet  or 
database.  Calculate  totals  if  comparable. 


Implementation: 

Until  there  is  a standardized  report  generation  system  that  captures  different  statistics  from  different  content 
providers,  we  recommend  that  each  library  develop  an  in-house  spreadsheet  or  database  to  capture 
monthly  usage  statistics  of  licensed  databases.  At  least  on  the  database  title  level,  usage  statistics  should 
be  collected  from  vendors,  entered  into  the  in-house  databank,  and  maintained  for  reporting  and  analysis. 


Collected  by:  Vendor 


Frequency:  Monthly,  but  can  be  reported  quarterly  or  annually 


Procedures: 

• Process  monthly  usage  statistics  from  vendors  and  copy  or  import  the  number  of  attempted  sessions  in 
each  database  (in  each  journal  collection  for  full-text  journals)  to  an  in-house  spreadsheet  or  database 
file. 

• Calculate  the  total  sessions  for  a given  month  by  adding  the  number  of  sessions  from  each  database  or 
journal  collection. 


Special  considerations:  Not  all  vendors  report  this  statistic.  Therefore,  it  will  be  necessary  to  qualify  the 
statistic  with  a sentence  such  as  this:  “We  have  150,000  logins  recorded  from  120  databases  out  of  200 
subscribing.  We  cannot  report  this  statistic  for  the  remaining  80  databases  because  the  vendor  does  not 
supply  login  (session)  information  to  customers.” 
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U3  - Number  of  Queries  (Searches) 
in  Electronic  Databases 


> Definition  - number  of  user  initiated  queries 
(searches)  in  licensed  electronic  resources. 

Usually  a search  is  recorded  each  time  a search 
request  is  submitted  to  the  server. 

> Rationale  - indicate  use  of  databases,  areas  of 
interest  to  clients,  level  of  use  beyond  individual 
session. 

> Related  Issues  - not  all  vendors  provide  this  data, 
need  standardization  of  what  is  counted  here  by 
different  vendors,  difficult  to  calculate  an  aggregate 
count. 
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Definition:  Number  of  user  initiated  queries  (searches)  in  licensed  electronic  resources.  A search  is  intended 
to  represent  a unique  intellectual  inquiry.  Typically,  a search  is  recorded  each  time  a search  request  is 
sent/submitted  to  the  server. 


Rationale:  This  statistic  provides  libraries  with  an  indication  of  the  databases  that  are  most  heavily  used, 
areas  of  user  interest,  database  popularity,  and  a level  of  usage  detail  that  goes  beyond  an  initial  session.  It 
also  can  provide  important  information  for  billing  purposes,  as  some  vendors  charge  for  database  usage  by 
number  of  searches.  This  statistic  can  complement  Ul,  the  number  of  electronic  reference  transactions,  as 
more  user  requests  bypass  staff  mediations.  Some  portion  of  this  statistic  is  also  analogous  to  in-library  use 
of  reference  sources. 


Related  issues:  Different  assumptions  about  and  mechanisms  for  collecting  search  counts  by  different 
vendors  are  potential  threats  to  the  combined  count. 
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U3  - Number  of  Queries  (Searches) 
in  Electronic  Databases 


> Unit  of  Measure  - count  of  search  requests. 

> Data  Source  - vendor  report. 

> Frequency  - daily,  monthly,  annually,  etc. 

> Process  - get  monthly  data  from  vendors, 
copy  or  parse  data  for  each  database  to  an  in- 
house  spreadsheet  or  database.  Calculate 
totals  if  comparable. 
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Implementation 


Collected  by:  Vendor 


Frequency:  Monthly,  but  can  be  reported  quarterly  or  annually 


Procedures:  Until  there  is  a standardized  report  generation  system  that  captures  different  statistics  from 
different  content  providers,  we  recommend  that  each  library  develop  an  in-house  spreadsheet  or  database 
to  capture  monthly  usage  statistics  of  licensed  databases.  Usage  statistics  need  to  be  collected  from 
vendors,  entered  into  the  in-house  databank,  and  maintained  for  reporting  and  analysis. 


• Process  monthly  usage  statistics  from  vendors  and  copy  or  import  the  number  of  attempted  searches  in 
each  database  to  an  in-house  spreadsheet  or  database  file. 

• Calculate  the  total  number  of  searches  for  a given  month  by  adding  the  number  of  searches  from  each 
database  or  journal  collection. 


Special  considerations:  Because  some  vendors  do  not  report  this  statistic,  it  will  be  necessary  to  qualify  the 
statistic  with  a sentence  such  as  this:  “We  have  150,000  searches  recorded  from  120  databases  out  of 
200  subscribing.  The  other  80  do  not  provide  this  statistic.” 
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U4  - Number  of  Items  Requested  in 
Electronic  Databases 

> Definition  - number  of  items  requested  in  all 
of  the  library’s  electronic  resources.  Can 
include  journal  articles,  e-books,  and  other 
type  of  materials  - may  be  citation,  abstract, 
TOC,  full-text. 

> Rationale  - circulation  count  for  electronic 
content  - equivalent  to  in-house  counts. 

> Related  Issues  - not  all  vendors  provide 
this  data,  need  standardization  of  what  is 
counted  here  by  different  vendors,  difficult  to 
calculate  an  aggregate  count. 
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Definition:  Number  of  items  requested  in  all  of  the  library’s  licensed  electronic  resources.  These  resources 
may  include  journal  articles,  e-books,  reference  materials,  and  non-textual  resources  that  are  provided  to  the 
library's  users  through  licensing  and  contractual  agreements.  The  user  requests  may  include  viewing, 
downloading,  emailing,  and  printing  to  the  extent  the  activity  can  be  recorded  and  controlled  by  the  server 
rather  than  browser. 

The  items  reported  depend  on  the  type  of  content.  Examples  include  citations,  abstracts,  tables  of  contents, 
and  full-text  articles  (ASCII,  HTML,  PDF,  or  PS). 


Rationale:  This  statistic  provides  a circulation  count  for  electronic  contents  in  a way  analogous  to  the 
traditional  circulation  of  books.  Given  the  fact  that  libraries  do  not  have  good  measurements  of  in-house 
materials  usage,  particularly  serials  usage,  this  statistic  helps  libraries  understand  in-library  use  patterns  that 
were  heretofore  difficult  to  measure. 


Related  issues:  Different  vendors  apply  different  assumptions  and  mechanisms  in  collecting  items 
requested  counts.  This  lack  of  standardization  makes  it  difficult  to  calculate  an  aggregate  count. 

We  do  not  have  good  measurement  of  in-house  materials  usage,  particularly  journal  usage.  However, 
electronic  journals  and  databases  allow  libraries  to  find  out  how  often  materials  are  requested.  Having  in- 
house  usage  figures  is  important  for  understanding  the  dynamics  of  usage  between  print  and  electronic 
journals,  so  that  we  can  ascertain  any  correlation  between  them. 
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U4  - Number  of  Items  Requested  in 
Electronic  Databases 


> Unit  of  Measure  - count  of  items  requested. 

> Data  Source  - vendor  report. 

> Frequency  - daily,  monthly,  annually,  etc. 

> Process  - get  monthly  data  from  vendors, 
copy  or  parse  data  for  each  database  to  an  in- 
house  spreadsheet  or  database.  Calculate 
totals  if  comparable. 
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Implementation 


Collected  by:  Vendor 


Frequency:  Monthly,  but  can  be  reported  quarterly  or  annually 


Procedures:  Until  there  is  a standardized  report  generation  system  that  captures  different  statistics  from 
different  content  providers,  we  recommend  that  each  library  develop  an  in-house  spreadsheet  or  database 
to  capture  monthly  usage  statistics  of  licensed  databases.  Usage  statistics  should  be  collected  from 
vendors,  entered  into  the  in-house  databank,  and  maintained  for  reporting  and  analysis. 


• Process  monthly  usage  statistics  from  vendors  and  copy  or  import  the  number  of  items  selected  for 
viewing,  downloading,  and  emailing  in  each  database.  Count  the  number  and  type  of  items  users 
selected:  abstracts,  citations,  and  full-texts. 

• Calculate  the  total  number  of  items  for  a given  month  by  adding  the  number  of  items  requested  from 
each  database  or  journal  collection. 


Special  considerations:  Because  some  vendors  do  not  report  this  statistic,  it  will  be  necessary  to  qualify  the 
statistic  with  a sentence  such  as  this:  “More  than  150,000  items  were  requested  from  120  databases  out 
of  200  subscribing.  The  other  80  do  not  provide  this  statistic.  Among  the  requested  items,  100,000  were 
some  form  of  full-text  records.” 
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U5  - Virtual  Visits  to  Library’s  Website  and 

Catalog 


> Definition  - number  of  client  visits  to  the  library’s 
Web  site  or  catalog  from  outside  the  physical 
library  premises  without  regard  to  the  number  of 
pages  viewed.  As  some  may  be  misleading,  this 
is  an  estimate  only. 

> Rationale  - reflects  external  interest  in  library 
services;  show  demand  for  library  resources;  can 
be  used  to  justify  investment  in  electronic 
resources. 

> Related  Issues  - need  staff  with  systems  skills. 


Definition:  This  is  defined  as  uservisits  to  the  library’s  Web  site  or  catalog  from  outside  the 
physical  1 ibrary  premises  regardless  of  the  number  of  pages  or  elements  viewed.  If  a user  looks  at 
16  pages  and  54  graphic  images  while  at  a Web  site,  that  user  registers  one  visit  on  the  Web 
server.  All  visits  to  the  Web  site  should  be  counted  regardless  of  repetition  by  one  user.  A visit  is 
usually  determined  by  a user’s  IP  address,  which  can  be  misleading  due  to  Internet  Service 
Providers  (ISPs)  and  Firewalls  or  Proxy  Servers.  Thus,  this  measure  is  actually  an  estimate  of  the 
visits. 


Rationale:  Use  of  the  Web  site  or  catalog  from  outside  the  library  reflects  interest  in  library 
services.  The  role  of  networked  services  is  to  expand  the  reach  of  libraries  beyond  their  physical 
boundaries.  This  statistic  helps  describe  the  significance  of  networked  services  use  by  measuring 
the  number  of  virtual  accesses.  This  will  also  give  an  opportunity  for  the  library  to  compare  the 
demand  placed  on  their  networked  resources  with  that  for  other  popular  information-oriented 
Web  sites  (such  as  Excite,  Lycos,  etc.). 


Related  issues:  This  measurement  requires  a relatively  high  degree  of  technical  skills  either  on 
staff  or  available  from  the  library’s  Web  site  host. 


OU5  - Virtual  Visits  to  Library's  Website  and 
Catalog 

> Unit  of  Measure  - count  visits  to  the  library’s  Web 
site  or  catalog . 

> Data  Source  - local  count  of  external  access  to 
library  Web  site,  remote  logins  to  non -Web  based 
library  databases  & remotely  accessible  library 
OPAC. 

> Frequency  - daily,  monthly,  annually,  etc. 

> Process  - identify  all  sources  of  visits  and  local 
servers;  exclude  internal  use;  use  log  analysis 
software  - possibly  parse  data  to  an  in-house 
spreadsheet  or  database. 


IS 


Implementation: 

Collected  by:  Local 

Frequency:  Reported  annually,  although  internal  reporting  will  be  more  frequent  (e.g.,  weekly,  monthly,  and  quarterly). 


Procedures: 


Identify  all  sources  of  virtual  visits  to  the  library.  This  may  involve  activities  that  take  place  on  more  than  one  Web 
server.  Some  of  the  Web  servers  may  be  owned  by  the  library  and  some  may  be  owned  or  maintained  by  another 
department  in  the  university,  an  Internet  Service  Provider  (ISP),  or  other  library  vendors  (e.g.,  library  OPAC 
provider). 


Exclude  internal  use  within  the  premises  of  the  library  from  the  counts  for  this  measure  when  possible.  Two  common 
approaches  are  using  IP  addresses  or  some  form  of  authentication  tagged  to  each  transaction.  In  terms  of  external 
visits  to  the  library,  three  common  sources  are:  external  access  to  the  library’s  Web  page,  remote  logins  (sessions)  to 
non- Web-based  library  databases,  and  remotely  accessible  library  OPAC. 

Develop  strategies  for  collecting  the  data  from  each  of  these  sources  of  virtual  visits.  Different  software  may  be 
needed  to  measure  each  electronic  source  of  virtual  visits.  In  some  cases,  the  library  may  calculate  the  virtual  visits 
using  one  or  more  log  analysis  software  packages.  In  other  cases,  the  external  owner  of  the  Web  server  or  service 
(the  ISP)  must  provide  the  data.  Discussions  may  need  to  be  held  with  these  service  providers  to  obtain  the  needed 
data.  In  still  other  cases,  custom  programs  may  have  to  be  developed. 


• In  the  case  of  library  Web  pages  housed  on  the  library  server,  identify,  configure,  and  install  appropriate  log  analysis 
software.  Determine  log  analysis  software  definition  that  corresponds  to  the  virtual  visit  definition. 


Note:  Different  log  analysis  software  packages  may  count  virtual  visits  in  different  ways,  so  the  count  obtained  will  by 
necessity  be  an  estimate.  Arrange  with  the  server  technical  staff  for  regular  (monthly)  reporting  of  internal  visits  at 
the  various  user  access  Internet  workstations,  external  library  user  virtual  visits,  and  total  virtual  visits  (internal  visits 
plus  external  visits).  Run  the  log  analysis  software. 

• In  the  case  of  library  Web  pages  housed  on  an  ISP's  server,  identify  the  log  analysis  software  the  ISP  uses.  Determine 
the  definition  of  “visit”  used  by  the  log  analysis  software  that  corresponds  to  the  virtual  visit  definition  with  the 
assistance  of  the  ISP.  Arrange  with  the  ISP  for  regular  (monthly)  reporting  of  internal  library  visits  at  the  various  user 
access  Internet  workstations,  external  library  user  virtual  visits,  and  total  virtual  visits  (internal  visits  plus  external 
visits). 


• Where  virtual  visit  counts  include  the  aggregate  of  internal  and  external  visits,  indicate  this  in  your  report 
Special  considerations:  Count  all  visits  to  the  Web  site  regardless  of  repetition  by  one  user  as  long  as  each  visit  meets 
the  criteria  for  this  statistic. 


After  one  user  connects  to  the  Internet,  several  users  could  conduct  multiple  different  searches  in  the  electronic  service. 

In  some  cases,  e.g.,  Internet-accessible  OPAC  use  inside  the  library,  several  users,  one  after  the  other,  might  make  use 
of  the  same  established  connection.  In  most  systems,  a connection  is  cut  off  after  a specified  period  of  non-use,  thus 
solving  part  of  the  problem.  The  best  existing  method  of  collecting  virtual  visits  is  to  use  log  analysis  software.  The 
log  analysis  software  producers  may  define  virtual  visits  differently.  For  example,  does  a visit  end  after  a time-out 
period  of  30  minutes,  15  minutes,  or  some  other  time?  The  recommended  time-out  period  is  30  minutes,  but  a local 
library  may  have  to  accept  the  available  log  analysis  software’s  definition  even  if  it  varies  from  the  above. 

Some  libraries  will  find  it  difficult  to  report  every  virtual  visit.  For  example,  libraries  may  have  difficulty  counting  the 
use  of  library  OP  ACS  because  their  vendors  do  not  provide  this  information.  Make  a record  of  those  sources  of 
virtual  visits  not  counted.  Do  not  estimate  virtual  visits  for  which  data  are  not  available. 
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Expenditures  for  Networked  Resources  & 
Related  Infrastructure 


> Cl  - Cost  of  electronic  full-text  journals 

> C2  - Cost  of  electronic  reference  sources 

> C3  - Cost  of  electronic  books 

> C4  - Library  expenditures  for 
bibliographic  utilities,  networks  & 
consortia 

> C5  - External  expenditures  for 
bibliographic  utilities,  networks  & 
consortia 


Statistics  Related  to  Expenditures  for  Electronic  Resources  and  Related  Infrastructure 

This  portion  of  the  statistics  is  based  on  the  ARL  Supplementary  Statistics  Survey  (the  most  recent  survey 
instruction  available  at  http://www.arl.Org/stats/aristat/#sup).  In  collecting  the  statistics,  the  library  should 
refer  to  the  procedures  followed  and  the  amounts  reported  in  response  to  the  ARL  Supplementary  Statistics 
Survey. 

These  statistics  were  developed  by  ARL  to  determine  expenditure  patterns  on  electronic  and  networked 
resources  and  the  effect  of  new  types  of  library  resources  and  services,  those  delivered  both  individually  and 
collectively  with  other  institutions,  on  library  expenditures.  These  measures  are  expected  to  help  ARL 
libraries  justify  their  growing  budgets  due  to  the  great  expense  of  electronic  and  networked  services.  These 
measures  can  help  answer  such  questions  as:  How  much  are  research  libraries  spending  for  electronic 
resources  collectively  and  how  much  on  average?  How  do  expenditures  for  electronic  resources  compare 
across  several  research  libraries? 

We  have  not  included  the  cost  of  the  technical  staff  and  their  training,  the  networking  and  equipment  to 
provide  access  to  the  electronic  resources  as  well  as  the  time  of  all  the  staff  involved.  This  will  have  to  be 
addressed  in  the  future. 


General  Introduction  to  C1-C3 

The  report  should  include  expenditures  for  electronic  indexes  and  reference  tools,  electronic  full-text 
periodical  collections  and  electronic  journal  back-files,  and  online  searches  of  remote  databases — whether 
accessed  remotely  or  installed  locally  from  CD-ROM,  magnetic  tapes,  etc.  The  report  should  also  include 
expenditures  for  materials  purchased  jointly  with  other  institutions  if  such  expenditures  can  be  separated 
from  other  charges  for  joint  services,  fees  paid  to  bibliographic  utilities  if  the  portion  paid  for  computer  files 
and  search  services  can  be  separately  counted,  and  equipment  costs  when  they  are  inseparably  bundled  into 
the  price  of  the  information  product 


Expenditures  for  bibliographic  utilities,  networks,  and  consortia  that  are  unrelated  to  end-user  database 
access  should  be  reported  in  C4,  not  in  C 1 through  C3. 
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Cl  - Cost  of  Electronic  Full-Text 
Journals 

> Definition  - expenditures  for  electronic  full-text 
journal  subscriptions.  Include  initial  purchase 
cost,  membership  fees,  annual  licenses  paid 
directly  or  as  part  of  consortia. 

> Rationale  - indicates  expenditures  (costs)  for  e- 
journals;  show  increased  demand  for  e-journals 
- which  replace  print;  can  calculate  unit  cost  of 
e-journal  by  collecting  Cl  & R1  statistics  - 
measure  of  effectiveness. 

> Related  Issues  - use  to  see  expenditure 
trends;  longitudinal  analysis. 


Definition:  Expenditures  for  electronic  full-text  journal  subscriptions  that  the  library  provides  to  its  users. 
Include  both  initial  purchase  cost,  membership  fees  (such  as  JSTOR)  as  well  as  annual  access  and  service 
fees  paid  directly  or  through  consortia  arrangements. 


Rationale:  This  statistic,  cost  of  electronic  full-text  journals,  was  developed  by  ARL  to  find  out  how  much 
libraries  are  spending  on  electronic  full-text  journals  and  how  new  forms  of  electronic  journals  are  replacing 
traditional  journals  and  scholarly  publications.  It  also  indicates  the  extent  of  budget  allocations  for  electronic 
resources.  Furthermore,  this  statistic  allows  libraries  to  calculate  unit  costs  of  e-journals  after  collecting  Cl 
and  R1  statistics,  and  thus  aids  libraries  in  deciding  how  effectively  they  are  serving  their  potential  and 
intended  audiences,  and  in  benchmarking  with  the  other  institutions. 


Special  considerations:  Whereas  the  patron  accessible  resource  counts  reflect  the  extensiveness  of 
electronic  resources  at  a given  point  in  time  (most  likely  at  the  end  of  the  reporting  period,  be  it  a month  or  a 
year),  the  cost  figures  cover  a period  of  time.  Ideally  the  amount  of  money  spent  reflects  the  number  and 
extensiveness  of  resources.  Prorating  licensing  fees  addresses  part  of  the  problem  of  matching  the  resources 
with  the  money  spent,  but  can  be  very  time  consuming.  You  can  report  the  annual  amount  paid  without 
prorating  on  the  basis  that  over  the  years  the  figures  will  even  out. 


Some  electronic  full-text  journals  come  either  as  a free  service  with  a print  subscription  or  as  part  of  a print- 
pi  us-on line- access  subscription  (the  library  pays  extra  for  electronic  access).  In  the  first  case,  the  problem  is 
whether  or  not  to  post  any  amount  for  the  cost  of  electronic  access.  In  the  latter  case,  the  question  is  how 
much  of  the  cost  can  be  attributed  to  electronic  access. 


347 


27 


Cl  - Cost  of  Electronic  Full-Text 
Journals 


> Unit  of  Measure  - cost  in  dollars. 

> Data  Source  - local  data  (acquisition 
report). 

> Frequency  - annually. 

> Process  - identify  all  expenditures  to  the 
lowest  level  - one  item;  can  group  by 
material  type,  vendor,  publisher,  subject, 
fund;  parse  data  to  an  in-house 
spreadsheet  or  database. 


Implementation 

Collected  by:  Local 

Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly,  quarterly) 

Procedures:  For  the  definition  of  electronic  full-text  journals,  please  refer  to  the  definition  of  Rl.  Current 
library  accounting  systems  do  not  support  coding  of  materials  expenditures  by  the  categories  used  in  the 
manual.  Therefore,  it  may  become  necessary  to  create  an  in-house  spreadsheet  or  database  file  to  keep 
track  of  cost  information  according  to  the  types  of  resources  (e-joumals,  reference  databases,  and  e- 
books).  Preferably  a single  file  will  contain  contract  information  (duration,  cost),  updated  title  counts, 
and  reported  usage  statistics.  Significant  coordination  is  required  for  setting  up  the  structure  of  the  file, 
but  in  the  long  run  may  streamline  many  aspects  of  the  management  of  electronic  licensed  materials. 

• Gather  reports  and  invoices  related  to  electronic  databases  and  resources  for  the  reporting  period.  These 
documents  are  typically  handled  by  the  library’s  accounting  office. 

• If  you  have  not  done  so,  organize  the  data  using  the  sample  worksheet  in  Appendix  C,  Figure  C.  1. 

• Even  if  a licensing  contract  or  consortium  arrangement  period  is  different  from  the  reporting  period,  use 
the  annual  licensing  fee  to  calculate  the  statistics. 

• If  a fee  is  paid  to  a consortium  or  other  joint  arrangement,  include  the  amount.  In  the  case  where  a fee  is 
paid  for  an  aggregate  service  and  the  service  contains  different  categories  of  resources  (full-text  journals 
and  reference  sources)  as  a bundle,  use  an  estimate  based  on  expected  or  historical  use,  or  list  prices. 

• Note  any  major  commitments  (such  as  JSTOR  one-time  costs)  that  do  not  occur  year  to  year  and  that 
significantly  influence  the  reported  amount 
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C2  - Cost  of  Electronic  Reference 
Sources 


> Definition  - expenditures  for  electronic  reference 
sources  & aggregate  services.  Include  annual 
access  fees  & other  service  costs  paid  directly  to 
vendor  or  through  consortia  arrangements. 

> Rationale  - indicates  expenditures  (costs)  for  e- 
reference  databases;  shows  shift  of  budget 
allocations  to  electronic  databases;  can  calculate 
unit  cost  of  database  by  collecting  C2  & R2 
statistics  - measure  of  effectiveness. 

> Related  Issues  - use  to  see  expenditure  trends; 
longitudinal  analysis. 
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Definition:  Expenditures  for  electronic  reference  sources  and  aggregate  services  that  the  library  provides  to 
users  either  through  individual  licensing  contracts  with  content  providers  or  through  consortia  or  other 
arrangements  where  the  library  pays  some  fees.  These  fees  include  both  annual  access  fees  and  other  service 
costs  paid  to  the  vendor  directly  or  through  consortia!  arrangements. 


Rationale:  This  statistic,  cost  of  electronic  reference  sources,  was  developed  by  ARL  to  determine  how 
much  libraries  are  spending  on  electronic  reference  sources  and  how  new  forms  of  electronic  reference 
sources  are  replacing  traditional  reference  materials.  It  also  gives  insight  into  shifts  in  budget  allocations 
from  print  to  electronic  materials,  or  new  allocations  exclusively  for  electronic  materials.  Furthermore,  this 
statistic  allows  libraries  to  calculate  unit  costs  of  electronic  reference  sources  after  collecting  C2  and  R2 
figures.  This  figure  assists  libraries  in  making  decisions  about  how  effectively  they  are  serving  their 
potential  and  intended  audience,  and  in  benchmarking  with  other  institutions. 


Special  considerations:  Whereas  the  patron  accessible  resource  counts  reflect  the  extensiveness  of 
electronic  resources  at  a given  point  in  time  (most  likely  at  the  end  of  reporting  period,  be  it  a month  or  a 
year),  the  cost  figures  cover  a period  of  time.  Ideally  the  amount  of  money  spent  reflects  the  number  and 
extensiveness  of  resources.  Prorating  licensing  fees  addresses  part  of  the  problem  of  matching  the  resources 
with  the  money  spent,  but  can  be  very  time  consuming.  You  can  report  the  annual  amount  paid  without 
prorating  on  the  basis  that  over  the  years  the  figures  will  even  out. 
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C2  - Cost  of  Electronic  Reference 
Sources 


> Unit  of  Measure  - cost  in  dollars. 

> Data  Source  - local  data  (acquisition  report). 

> Frequency  - annually. 

> Process  - definition  of  electronic  ref  resource  - 
see  R2.  Can  group  by  individual  title,  vendor, 
publisher,  subject,  fund;  parse  data  to  an  in-house 
spreadsheet  or  database. 
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Implementation 


Collected  by:  Local 


Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly,  quarterly) 


Procedures:  For  the  definition  of  electronic  reference  sources,  please  refer  to  the  definition  of  R2.  For 
libraries  that  do  not  have  acquisitions  systems  which  support  coding  of  materials  expenditures  by  the 
categories  used  in  the  manual,  it  may  be  necessary  to  create  an  in-house  spreadsheet  or  database  file  to 
keep  track  of  cost  information  according  to  the  types  of  resources  (e-journals,  reference  databases,  and  e- 
books).  Preferably  a single  file  will  contain  contract  information  (duration,  cost),  updated  title  counts, 
and  reported  usage  statistics. 


• Gather  reports  and  invoices  related  to  electronic  databases  and  resources  for  the  reporting  period.  These 
documents  are  typically  handled  by  the  library's  accounting  office. 

• If  you  have  not  done  so,  organize  the  data  using  the  sample  worksheet  in  Appendix  C,  Figure  C.  1 . 

• Even  if  a licensing  contract  or  consortium  arrangement  period  is  different  from  the  reporting  period,  use 
the  annual  licensing  fee  to  calculate  the  statistics. 

• If  a fee  is  paid  to  a consortium  or  through  other  joint  arrangement,  include  the  amount.  If  a fee  is  paid 
for  an  aggregate  service  and  the  service  contains  different  categories  of  resources  (full-text  journals  and 
reference  sources)  as  a bundle,  use  an  estimate  based  on  expected  or  historical  use,  or  list  prices. 

• In  the  comments  field  of  the  sample  worksheet  (Appendix  C,  Figure  C.l),  report  any  major 
commitments  that  do  not  occur  year  to  year  and  that  significantly  influence  the  reported  amount. 
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C3  - Cost  of  Electronic  Books 


Definition  - expenditures  for  electronic  full-text 
monographs.  Include  annual  purchase  costs  & 
membership  fees,  annual  access  & service  fees 
paid  directly  to  vendor  or  through  consortia 
arrangements. 

Rationale  - indicates  expenditures  (costs)  for 
e-books;  show  shift  of  budget  allocations  to 
electronic  resources  - which  replace  print;  can 
calculate  unit  cost  of  e-book  by  collecting  C3  & 
R3  statistics  - measure  of  effectiveness. 

Related  Issues  - still  in  flux;  use  to  see 
expenditure  trends;  longitudinal  analysis. 

^prir  m w*  ^ 
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Definition:  Expenditures  for  electronic  full-text  monographs  that  the  library  offers  to  its  users.  Include  both 
initial  purchase  costs  and  membership  fees  as  well  as  annual  access  and  service  fees  paid  directly  or  through 
consortia  arrangements. 


Rationale:  This  statistic,  cost  of  electronic  books,  was  developed  by  ARL  to  determine  how  much  libraries 
were  spending  on  electronic  books.  It  also  gives  an  idea  about  the  extent  of  budget  allocations  for  electronic 
resources.  Furthermore,  this  statistic  allows  libraries  to  calculate  unit  costs  of  e-books  after  collecting  C3 
and  R3  statistics,  aids  them  in  determining  how  effectively  they  are  serving  their  potential  and  intended 
audiences,  and  assists  them  in  benchmarking  with  other  institutions. 


Related  issues:  In  many  instances,  the  physical  form  of  the  material  (print,  electronic)  may  change  the 
nature  of  the  object.  An  electronic  book  is  a good  example.  With  enhancements  such  as  full-text  searching 
(although  print  books  too  have  some  search  capability  through  tables  of  contents  and  indexes),  electronic 
books  support  new  forms  of  searching  not  present  in  print. 
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C3  - Cost  of  Electronic  Books 

> Unit  of  Measure  - cost  in  dollars. 

> Data  Source  - local  data  (acquisition  report). 

> Frequency  - annually. 

> Process  - definition  of  electronic  books  see 
R3.  Can  group  by  individual  title,  vendor, 
publisher,  subject,  fund;  parse  data  to  an  in- 
house  spreadsheet  or  database. 
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Implementation 

Collected  by:  Local 

Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (monthly,  quarterly) 

Procedures:  For  the  definition  of  electronic  books,  please  refer  to  the  definition  of  R3.  Current  library 
accounting  systems  generally  do  not  support  coding  of  materials  expenditure  by  the  categories  used  in 
the  manual.  [See  note  for  3. 2.4. 5]  Therefore,  it  may  become  necessaiy  to  create  an  in-house  spreadsheet 
or  database  file  to  keep  track  of  cost  information  according  to  the  types  of  resources  (e-journals, 
reference  databases,  and  e-books).  Preferably,  a single  file  will  contain  contract  information  (duration, 
cost),  updated  title  counts,  and  reported  usage  statistics. 


• Gather  reports  and  invoices  related  to  electronic  databases  and  resources  for  the  reporting  period.  These 
documents  are  typically  handled  by  the  library’s  accounting  office.  You  may  also  need  to  review 
circulation  records  to  verify  the  accuracy  of  invoices  if  additional  per-use  fees  are  paid  (royalty  on  use, 
as  with  E-Reserves). 

• If  you  have  not  done  so,  organize  the  data  using  the  sample  worksheet  in  Appendix  C,  Figure  C.  1 . 

• Even  if  a licensing  contract  or  consortium  arrangement  period  is  different  from  the  reporting  period,  use 
the  annual  licensing  fee  to  calculate  the  statistics. 

• If  a fee  is  paid  to  a consortium  or  other  joint  arrangement,  include  the  amount. 

• Note  any  major  commitments  (such  as  netLibrary  purchase  costs)  that  do  not  occur  year  to  year  and  that 
significantly  influence  the  reported  amount 

Special  considerations:  Whereas  the  patron  accessible  resource  counts  reflect  the  extensiveness  of  electronic 
resources  at  a given  point  in  time  (most  likely  at  the  end  of  reporting  period,  be  it  a month  or  a year),  the 
cost  figures  cover  a period  of  time.  Ideally  the  amount  of  money  spent  reflects  the  number  and 
extensiveness  of  resources.  Prorating  licensing  fees  addresses  part  of  the  problem  of  matching  the 
resources  with  the  money  spent,  but  it  can  be  very  time  consuming.  You  can  report  the  annual  amount 
paid  without  prorating  with  the  rationale  that  over  the  years  the  figures  will  even  out. 

Traditionally  books  are  purchased  on  a one-time  payment  in  exchange  for  permanent  ownership  by  the 
library.  However,  with  regard  to  electronic  books,  it  appears  that  some  arrangements  allow  libraries  to 
subscribe  to  an  e-book  collection  at  a predetermined  fee  and  for  a predetermined  interval  of  time.  We 
are  concerned  with  the  format  of  the  material,  not  the  subscription  or  payment  arrangement.  These 
materials  should  be  counted  as  books,  not  serial  publications. 
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Sample  E-Resource  Cost  Report  Form 
Figure  C.1  (forC1-C3) 


Reporting  Period:. 
Name  of  Library:_ 


Resource/  Consortium  Name  and  Type; 

yj§ij|  ' FulMext  journals ' <.&)% 

Ite^niclwoks  (3)" 

' r . ■ ,•  \ i/  ^ 

v ;>  ;1-  ;Comments  - 'i?< ; 

Sub  Total  (1) 

Sub  Total  (2) 

Sub  Total  (3) 

Grand  Total  (1+2*3) 

mv  ■ A w&>*. 

353.;; 


C4  - Library  Expenditures  for 
Bibliographic  Utilities,  Networks  & 
Consortia 

Definition  - expenditures  for  services  provided  by 
national,  regional  & local  bibliographic  utilities, 
networks,  & consortia  (OCLC,  RLG);  exclude  fees 
paid  for  client  database  access  which  should  be 
reported  in  Cl  through  C3. 

Rationale  - opportunity  for  benchmarking  by  file  size; 
shows  in-house  digitizing  effort.  Shows  extent  of 
digital  library  projects,  resource  & “virtual  storage" 
requirements. 

Related  Issues  - each  digital  collection  is  unique; 
important  to  use  appropriate  units  of  measure  to 
describe  overall  size  & extensiveness  of  collection. 


Definition:  Expenditures  paid  by  the  library  for  services  provided  by  national,  regional,  and  local 
bibliographic  utilities,  networks,  and  consortia  such  as  OCLC,  RLG,  excluding  fees  paid  for  user  database 
access  and  subscriptions,  which  should  be  reported  in  Cl  through  C3. 


Rationale:  This  statistic  is  based  on  the  ARL  Supplementary  Statistics  Survey.  It  was  developed  by  ARL  to 
determine  how  much  money  libraries  spend  for  bibliographic  utilities,  networks,  and  consortia.  Because 
individual  libraries  often  have  to  deal  with  special  provisions  and  funding  issues  related  to  contracts,  this 
statistic  may  not  laid  itself  to  comparability  among  ARL  member  libraries.  Nevertheless,  it  represents  an 
attempt  to  keep  track  of  the  financial  relationships  between  bibliographic  utilities  and  libraries.  Although 
this  may  provide  very  limited  comparability,  it  is  an  estimate  of the  cost  of  bibliographic  utilities,  networks, 
and  consortia. 


Special  considerations:  Prorating  can  be  time  consuming.  Consortia  or  other  memberships  may  bring 
additional  benefits,  such  as  subscriptions,  training,  or  preferential  pricing  for  acquisition  of  materials.  It  may 
be  difficult  to  separate  pure  membership  fees  from  value-added  services  of  membership  (e.g.,  original 
catalog  credits  from  OCLC  that  may  be  used  to  offset  costs  of  databases,  purchase  of  catalog  records,  etc.). 
Report  the  annual  amount  paid  without  prorating  with  the  rationale  that  over  the  years  the  figures  will  even 
out. 
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C4  - Library  Expenditures  for 
Bibliographic  Utilities,  Networks  & 
Consortia 


> Unit  of  Measure  - cost  in  dollars. 

> Data  Source  - local  data  (acquisition  report). 

> Frequency  - annually. 

> Process  - data  from  acquisitions  system; 
parse  data  to  an  in-house  spreadsheet  or 
database.  Sample;  parse  data  to  an  in-house 
spreadsheet  or  database. 
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Implementation 


Collected  by:  Local 


Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly,  quarterly) 


Procedures: 

• Gather  reports  and  invoices  with  bibliographic  utilities,  networks,  and  consortia  of  which  the  library  is  a 
member  for  the  whole  or  part  of  the  reporting  period.  These  documents  are  typically  handled  by  the 
library’s  accounting  office. 

• Identify  only  those  expenditures  paid  to  the  bibliographic  utilities,  networks,  and  consortia  for 
membership,  maintenance,  and  other  infrastructure.  Do  not  include  expenditures  that  are  directly 
attributable  to  access  of  electronic  resources.  Those  expenditures  should  be  included  in  Cl  through  C3. 
For  instance,  if  your  library  paid  a total  of  $100,000  to  OCLC  for  its  various  services  and  your  best 
guess  of  electronic  database  access  portion  of  the  services  is  80%,  then  you  should  report  $80,000  for 
C2  and  the  remaining  $20,000  for  C4. 

• Even  if  a membership  or  consortium  period  is  different  from  the  reporting  period,  use  the  amount  of  the 
membership  or  consortium  agreement. 

• Use  the  sample  form  in  Appendix  C,  Figure  C.2  to  compile  the  expenditures. 
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Sample  Consortia  Expenditure  Report  Form 
Figure  C.2  (for  C4) 


Reporting  Period: 
Name  of  Library 


f Comments 

ERIC 
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C5  - External  Expenditures  for 
Bibliographic  Utilities,  Networks  & 
Consortia 

> Definition  - expenditures  paid  by  external 
agencies,  on  the  library’s  behalf,  for  access 
to  computer  files,  e-journals  or  search 
services  through  a centrally  funded  system 
or  consortia  arrangements.  (Examples  - 
VIVA  (Virginia),  CNSLP  (Canada),  U-Cal 
California  Digital  Library  Expenditures). 

> Rationale  - based  on  the  ARL 
Supplementary  Statistics.  Estimate  of 
external  costs  that  may  become 
expenditures  eventually  when  funds  run  out. 
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Definition:  Expenditures  paid  by  external  agencies,  such  as  state  government  agencies,  on  the  library’s 
behalf  for  access  to  computer  files,  electronic  serials,  or  search  services  through  a centrally  funded  system  or 
consortial  arrangements.  Examples  include  state-  (or  province-)  supported  networks  such  as  VIVA 
(Virginia),  CNSLP  (Canadian  National  Site  Licensing  Project),  and  the  University  ofCalifornia's  California 
Digital  Library  Expenditure. 


Rationale:  Like  statistic  C4,  this  statistic  is  based  on  the  ARL  Supplementary  Statistics  Survey.  It  was 
developed  by  ARL  to  determine  how  much  money  is  spent  for  bibliographic  utilities,  networks,  and 
consortia  on  libraries'  behalf  for  access  to  computer  files,  serials,  and/or  services  through  consortial 
arrangements.  Because  of  contractual  issues,  this  statistic  may  provide  little  comparability  among  ARL 
member  libraries.  Nevertheless,  it  can  give  ARL  members  an  estimate  of  the  external  costs  of  bibliographic 
utilities,  networks,  and  consortia. 


ERIC 
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C5  - External  Expenditures  for 
Bibliographic  Utilities,  Networks  & 
Consortia 


> Unit  of  Measure  - cost  in  dollars. 

> Data  Source  - local  data  (acquisition  report). 

> Frequency  - annually. 

> Process  - data  from  acquisitions  system  or 
collections  office  or  find  out  which  part  of  the 
funding  is  attributed  to  local  library;  create  an 
in-house  spreadsheet  or  database. 


r-7- — "nrr; — ■» — 
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Implementation 


Collected  by:  Local  and  external  bodies  such  as  regional  and  academic  consortia 


Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly,  quarterly). 


Procedures: 

• Gather  reports  and  invoices  with  bibliographic  utilities,  networks,  and  consortia  that  are  related  to 
electronic  databases  and  resources  for  the  reporting  period.  These  documents  are  typically  handled  by 
the  library’s  accounting  office.  However,  they  can  be  maintained  outside  the  organization  and,  in  some 
instances,  may  only  be  provided  to  libraries  upon  demand. 

• Find  out  how  much  of  the  central  funding  is  attributable  to  your  library.  For  example,  if  your  library 
contributes  a total  of  $60, 000  over  a period  of  three  years  to  a state  consortium  that  has  a matching 
contribution  of  $1 20,000  for  the  same  period,  the  amount  to  report  as  C5  for  a given  year  during  the 
three-year  period  will  be  $40,000  ($120,000  x 1/3).  The  library’s  contribution  ($60,000)  has  to  be 
divided  annually  and  posted  in  Cl  through  C3. 

• If  the  specific  dollar  amount  is  not  known,  but  the  total  student  FTE  for  the  consortium  and  the  amount 
spent  for  the  academic  members  are  known,  divide  the  overall  amount  spent  by  your  institution’s  share 
of  the  total  student  FTE.  Alternatively,  if  the  consortium  is  comprised  of  different  types  of  institutions 
(academic,  public,  or  corporate),  but  the  library  has  information  about  the  portion  of  its  own  use  among 
the  consortium  participants,  multiply  the  total  amount  by  the  percentage  of  known  (or  estimated)  usage 
rate. 

• As  a last  resort,  consult  with  a staff  member  overseeing  the  consortium  or  the  central  funding  system  to 
get  an  estimate  of  the  portion  of  the  central  funding  that  is  attributable  to  the  library.  Please  make  a note 
of  this  in  the  comments  field  in  the  sample  worksheet  (Appendix  C,  Figure  C.3). 

• Use  the  sample  form  in  Appendix  C Figure  C.3  to  compile  the  expenditures. 
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Sample  Consortia  Funding  Report  Form 
Figure  C.3  (for  C5) 


Reporting  Period:_ 
Name  of  Library:_ 


/ Consortia  Name 

* * ik?  *'*V  ■ ^ 

v,’  Total  Fuhdlngiv 

. -» 11  ' . . " 

Amount  J§V 
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BEST  COPY  AVAILABLE 
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Library  Digitization  Activities 


> D1  - Size  of  library  digital  collection 

> D2  - Use  of  library  digital  collection 

> D3  - Cost  of  digital  collection  construction 
& management 

(Collecting  these  data  requires  staff  familiar  with  the 


- 

digital  environment.) 

XMM 
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4.3.4  Statistics  Related  to  Library  Digitization  Activities 

Comprised  of  resource  and  use  measures,  the  digital  collection  measures  attempt  to  describe  where  libraries 
are  in  creating  and  making  available  local  (perhaps  unique)  content  that  may  not  have  been  previously 
accessible.  Such  collections  can  attract  students  and  faculty  to  your  university  and  thereby  enhance  the 
institution’s  reputation.  As  more  libraries  digitize  resources,  more  users  will  be  able  to  retrieve  those  unique 
resources  at  anytime  and  from  anywhere.  Digital  library  projects,  as  well  as  other  network  resources  and 
services,  also  will  serve  increasing  numbers  of  students  taking  courses  online. 

Collecting  library  digitization  measures  may  provide  an  opportunity  for  benchmarking  and  may  encourage 
libraries  to  devote  more  time  and  allocate  more  resources  to  this  worthwhile  endeavor.  It  should  be  noted 
that  these  statistics  represent  a very  early  attempt  to  measure  digitization  of  resources;  as  time  passes  and  the 
technology  advances,  some  of  the  definitions  and  procedures  may  need  to  be  revisited  and  modified.  During 
the  field-testing  it  was  reported  that  storing  and  maintaining  digitized  resources  had  been  an  issue.  The 
unavailability  of  an  appropriate  infrastructure  in  some  institutions  meant  that  the  project  did  not  include 
statistics  related  to  library  digitization  projects. 

Libraries  archive  the  scholarly  output  of  their  institutions  — theses  and  dissertations — in  both  paper  and 
digital  form.  Digital  collections  also  provide  new  opportunities  with  faculty  to  archive  research  results. 
These  statistics,  although  preliminary,  form  a basis  for  tracking  these  issues. 


3G0 


D1  - Size  of  Library  Digital  Collection 


> Definition  - digital  materials  created  or  converted  by  the 
library  & made  available  electronically.  Includes  e-theses, 
special  collections,  maps,  sound  recordings,  films  - not 
purchased. 

> Rationale  - Collecting  library  digitization  measures  may 
provide  an  opportunity  for  benchmarking  in  terms  of  file  sizes 
for  the  resources  that  have  been  digitized.  Moreover,  the 
statistic  can  demonstrate  that  libraries  are  not  merely  brokers  of 
external  information  resources,  but  also  producers  of 
information  content  and  useful  finding  aids. 

This  statistic  provides  information  on  the  extent  of  digital  library 
projects,  the  life  cycle  of  such  projects,  and  the  "virtual  space" 
requirements  of  such  collections. 

> Related  Issues  - use  to  see  expenditure  trends; 
longitudinal  analysis. 
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Definition:  Library  digital  collection  refers  to  digital  materials  (texts,  images,  and  audio-visuals)  created  in 
or  converted  from  different  formats  (e.g.,  paper,  microfilm,  tapes,  etc.)  by  the  library  and  made  available  to 
users  electronically.  This  includes  electronic  theses  and  dissertations  (ETDs),  special  collections  materials, 
maps,  sound  recordings,  films,  and  other  digital  materials  that  are  not  purchased  or  acquired  from  outside 
through  individual  or  consortial  licensing  agreements.  Includes  the  number  of  titles  and  size  (in  gigabytes) 
by  sub-categories  (ETD,  visual  materials,  texts,  multimedia),  and  as  an  aggregate  at  the  end  of  the  reporting 
period.  Also  includes  the  number  of  items  (titles)  added  during  the  reporting  period. 


The  types  of  formats  in  Appendix  C,  Figure  C.4,  refer  to  original  formats  rather  than  the  digitized  outputs. 
Examples  of  visual  materials  include  photos,  maps,  and  postcards.  Examples  of  text  include  books,  journal 
articles,  and  pamphlets.  Examples  of  multimedia  include  audio,  video,  and  other  interactive  materials. 
However,  this  statistic  does  not  include  any  back  up  copies  or  mirror  sites  because  items  should  be  counted 
only  once. 


Rationale:  Collecting  library  digitization  measures  may  provide  an  opportunity  for  benchmarking  in  terms 
of  file  sizes  for  the  resources  that  have  been  digitized.  Moreover,  the  statistic  can  demonstrate  that  libraries 
are  not  merely  brokers  of  external  information  resources,  but  also  producers  of  information  content  and 
useful  finding  aids. 


This  statistic  provides  information  on  the  extent  of  digital  library  projects,  the  life  cycle  of  such  projects,  and 
the  "virtual  space"  requirements  of  such  collections. 


Related  issues:  Realistically,  each  digital  collection  is  unique  in  terms  of  the  production  process,  the  way  it 
is  intended  to  be  used,  its  focus,  and  maintenance.  It  is  important  to  use  appropriate  units  of  measurement  to 
describe  the  overall  size  and  extensiveness  of  the  whole  collection. 


Because  of  the  wide  variations  of  the  types  and  features  of  digital  collections  constructed  at  ARL 
institutions,  this  statistic  may  be  more  useful  locally  than  for  comparison  across  ARL  member  libraries. 
Benchmarking  may,  however,  be  possible  from  the  data  collected  to  produce  some  qualitative  and 
quantitative  indicators  as  to  the  extent  of  digital  library  collection  activities  and  different  emphases  across 
the  ARL  membership. 
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D1  - Size  of  Library  Digital  Collection 


> Unit  of  Measure  - title  count  & file  size  by  item 
type;  added  titles  by  reporting  period. 

> Data  Source  - local  data  (acquisition  report). 

> Frequency  - annually. 

> Process  - depending  on  digital  formats 
(derivatives);  pay  attention  to  formats  - count  file 
size.  For  inventory  use  Sample  tally. 
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Implementation 


Collected  by:  Local 


Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly,  quarterly) 

Procedures:  For  cases  in  which  multiple  digital  formats  (derivatives)  were  produced  from  an  item,  count  it 
only  once  based  on  the  type  of  item  that  was  digitized.  For  example,  if  a 100-page  book  was  digitized  in 
100  TIFF  files,  each  containing  a page,  a 100-page  PDF  file,  and  10  PDF  files  (one  PDF  file  for  each  of 
10  chapters),  count  it  as  a single  text  with  1 00  pages.  If  a derivative  item  was  used  as  the  source,  do  not 
count  the  outputs.  But  in  the  total  size  (in  gigabytes)  include  all  versions  of  derivatives. 


• Designate  a staff  member  to  coordinate  the  collection  of  this  statistic.  The  person  should  be  well  aware 
of  library  digital  collection  activities. 

• Identify  library  staff  in  charge  of  various  digital  library  projects  and  initiatives. 

• It  is  necessary  to  conduct  an  inventory  of  digital  material  stock  using  the  sample  tally  worksheet  in 
Appendix  C,  Figure  C.6  if  it  has  not  been  done  already.  If  this  inventory  information  is  already 
available,  enter  it  into  the  worksheet.  When  the  inventory  is  completed,  summarize  the  information 
using  the  sample  worksheet  in  Appendix  C,  Figure  C.4.  Add  additional  categories  if  necessary. 

• After  obtaining  the  inventory  information,  ask  staff*  members  to  keep  track  of  additional  output  regularly 
using  the  sample  tally  worksheet  in  Appendix  C,  Figure  C.6. 

• At  the  end  of  the  reporting  period,  collect  the  worksheets  and  calculate  the  total  production  during  the 
reporting  period  using  the  worksheet  in  Appendix  C,  Figure  C.5.  Add  additional  categories  if  necessary. 
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Sample  Library  Digital  Collection 
Report  Form  - Figure  C.6  (for  D1) 


Reporting  Period:. 


Name  of  Library:. 
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• #■ 

Tnfom»^ri|T^e 

,i.  B3T  3s;  .•*; 

3.  Texts  ;ii. 

|K  AudloAfldeo/Mu  itfmed  te 

^fjp. 

Location 

. ,8be(GB)  | 

fc, 

P*  : m 

43 


BEST  COPY  AVAILABLE 
363  1 


43 


Sample  Library  Digital  Collection  Inventory 
Report  Form  - Figure  C.4  (for  D1) 


Reporting  Period:. 
Name  of  Library 
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Titles 
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Size  (GB) 
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Sample  Library  Digital  Collection  Items  Added 
Report  Form  - Figure  C.5  (for  D1) 


Reporting  Period:. 
Name  of  Library:_ 


BEST  COPY  AVAILABLE 
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D2  - Use  of  Library  Digital  Collection 


> Definition  - number  of  times  digital  collection 
titles  and  files  are  accessed;  number  of  searches 
conducted  during  reporting  period. 

> Rationale  - as  these  collections  are  unique,  the 
information  is  used  locally.  May  show  quality  of 
local  service. 

> Related  Issues  - exclude  accesses  by  Web 
search  spiders  if  possible. 
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Definition:  Number  of  times  library  digital  collection  titles  and  physical  files  were  accessed  and  the  number 
of  searches  (queries)  conducted  (if  there  is  such  a capability)  during  the  reporting  period. 


Rationale:  Each  digital  collection  is  unique  in  terms  of  its  focus,  production  process,  and  the  ways  it  is 
intended  to  be  used  and  maintained.  Therefore,  because  of  the  wide  variations  of  the  types  and  features  of 
these  library  collections  constructed  at  different  ARL  institutions,  this  statistic  needs  to  be  collected  and  used 
locally  instead  of  across  ARL  member  libraries.  Nevertheless,  this  statistic  has  the  potential  to  produce  some 
qualitative  and  quantitative  indicators  as  to  how  these  collections  are  being  used  and  serving  the  intended 
user  community's  needs. 


Related  issues:  This  statistic  needs  to  be  collected  and  used  locally  instead  of  across  ARL  member  libraries 
because  of  the  wide  variations  of  the  types  and  features  of  digital  collections  constructed  at  ARL  institutions. 


36G 

o 

ERIC 


46 


Ini  piemen  tilt  ion 


D2  - Use  of  Library  Digital  Collection 


> Unit  of  Measure  - title  access  count;  search 
count. 

> Data  Source  - local  data  (OPAC  or  unique 
database). 

> Frequency -annually. 

> Process  - analyze  Web  logs  and  data  provided  by 
the  software.  May  need  knowledge  of  SQL  May 
collect  physical  file  access  count  only  to  save  time. 
May  install  special  Web  traffic  analysis  software 
(WfebTrends).  May  collect  only  sample  data  & 
extrapolate. 
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Collected  by:  Local 

Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly,  quarterly) 


Procedures: 


• Designate  a staff  member  to  coordinate  the  collection  of  this  statistic.  The  person  should  be  well  versed 
in  the  use  of  Web  log  software  and/or  statistics  provided  by  the  software.  This  person  will  act  as  a 
liaison  for  staff  members  who  are  responsible  for  managing  library  digital  collections.  Obtaining  the 
statistic  may  require  some  level  of  programming  (e.g.,  Unix  scripting  and  SQL). 

• Items  accessed  can  be  collected  in  various  ways,  and  depending  on  your  library’s  environment,  your 
library  may  need  to  collect  different  access  statistics. 

• Although  you  are  asked  to  collect  both  title  access  and  physical  file  access,  if  it  takes  too  much  time  and 
effort  to  collect  the  title  access,  report  the  physical  file  access  count  only.  For  example,  a book  can  be 
digitized  and  made  into  10  PDF  files,  each  containing  a chapter,  for  access.  Suppose  a user  viewed  five 
PDF  files  out  of  10.  In  this  case,  you  will  have  five  physical  item  accesses  and  one  title  access.  Usually 
it  is  easier  to  have  a physical  item  access  count,  while  it  takes  custom  programming  to  compute  the  title 
access  count  as  most  off-the-shelf  Web  traffic  software  packages  do  not  provide  this. 

• Do  not  report  Web  page  hits.  Instead,  count  how  many  times  the  digitized  items  were  accessed  (the 
exact  name  for  item  access  may  vary  depending  on  the  type  of  Web  traffic  analysis  software  being  used 
in  the  library). 

• If  a search  capability  is  a feature  of  a library  digital  collection,  the  total  number  of  searches  submitted 
needs  to  be  collected.  A search  represents  an  explicit  user  request  for  specific  information  in  a database 
and  is  expressed  usually  in  the  form  of  word  strings.  Clicks  on  Web  page  buttons,  such  as  “Next”  and 
“Previous,”  do  not  count  as  user  searches. 

• You  might  want  to  install  Web  traffic  analysis  software  (e.g.,  WebTrends,  Web  Tracks)  on  the  library 
Web  servers  housing  library  digital  collection  materials,  if  the  Web  servers  do  not  have  such  software 
already.  You  might  want  to  consider  installing  atrial  version  that  gives  between  30-180  days  of  free 
trial. 

• Read  the  description  of  reported  statistics  carefully  and  make  sure  that  the  software  provides  what  you 
want. 

• If  continuous  collection  of  use  statistics  is  not  possible  or  desirable,  select  a typical  week  (or  month)  to 
run  a sample  study.  Be  sure  to  vary  the  specific  week  (or  month)  chosen  over  the  course  of  a year  or 
from  year  to  year  to  account  for  seasonal  fluctuations.  Extrapolate  based  on  the  sample  data. 

• At  the  end  of  the  report  period,  use  the  log  analysis  report  to  calculate  the  number  of  accesses  to  library 
digital  collection  items.  Use  the  sample  report  in  Appendix  C,  Figure  C.7,  to  organize  the  information. 

Special  considerations:  To  the  extent  possible,  exclude  accesses  by  Web  search  spiders.  Also,  do  not 

include  accesses  to  auxiliaiy  (or  incidental)  items  that  are  not  part  of  the  library  digital  collection  content 
(.gif  buttons  and  image  maps  for  navigation).  Note  the  method  used  and  include  a description  of  any 
filtering  done. 
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Sample  Digital  Collection  Access 
Report  Form  - Figure  C.7  (for  D2) 


Reporting  Period:. 
Name  of  Library:_ 


Project 

Name 

Server 

Name 

Directory. 

Location 

tw'F'; 

Access 
: ; Count  / 

Item 

Access 

Count 

t&r 

Searches 

Comments 
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D3  - Cost  of  Digital  Collection 
Construction  and  Management 


> Definition  - direct  costs  (personnel,  equipment, 
software,  contracted  services)  to  create  digital 
materials  or  to  convert;  include  expenditures 
related  to  digitization,  OCR,  any  creation, 
preparation,  data  storage  & copyright  clearance; 
exclude  costs  for  resources  purchased 
externally. 


> Rationale  - as  these  collections  are  unique,  the 
information  is  used  locally  - may  show  quality  of 
local  service. 
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Definition:  Annual  direct  costs  (personnel,  equipment,  software,  contracted  services,  and  similar  items) 
spent  to  create  digital  materials  (texts,  images,  and  multimedia)  or  to  convert  existing  materials  into  digital 
form  for  the  purpose  of  making  them  electronically  available  to  users.  Include  expenditures  related  to 
digitization,  OCR,  editorial,  creation  of  markup  texts,  preparation  of  metadata  for  access  to  digitized 
materials,  data  storage,  and  copyright  clearance.  Exclude  expenditures  for  information  resources  purchased 
or  acquired  from  outside  the  institution  through  individual  or  consortial  licensing  agreements. 


Rationale:  The  cost  of  each  digital  collection  construction  may  vary  significantly,  depending  on  the  size  of 
the  collection,  conditions  of  the  sources  before  digitizing,  available  infrastructure,  staff  allocation,  timeline, 
and  administrative  support.  This  statistic  should  be  collected  and  used  locally  instead  of  across  ARL 
member  libraries  because  of  the  wide  variability  among  these  library  collections  constructed  at  different 
ARL  institutions.  Nevertheless,  this  statistic  has  the  potential  to  provide  quantitative  indicators  as  to  how 
costly  these  efforts  are,  how  much  resource  allocation  (i.e.,  budget  allocation,  staffing,  infrastructure,  etc.)  is 
needed,  and  how  well  they  serve  the  intended  user  community’s  needs  (e.g.,  to  account  for  internal  and 
external  costs  to  construct  and  manage  digital  collections  at  ARL  libraries). 


Related  issues:  This  statistic  needs  to  be  collected  and  used  locally  instead  of  across  ARL  member  libraries 
because  of  the  wide  variations  of  the  types  and  features  of  digital  collections  constructed  at  ARL  institutions. 
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D3  - Cost  of  Digital  Collection 
Construction  and  Management 


> Unit  of  Measure  - all  costs  - see  above. 

> Data  Source  - local  data  (acquisition  or  project 
office);  salaries  - from  accounting  or  personnel 
department. 

> Frequency -annually. 

> Process  - all  staff  involved  in  the  digital  initiative 
fill  out  worksheet.  For  reporting  period;  estimate 
time  spent  doing  work  - capture  as  FTE. 
Breakdown  of  activity  may  be  very  detailed;  salary 
information  calculated  & added;  cost  of  equipment 
amortized. 


Implementation 


Collected  by:  Local 


Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent  (e.g.,  monthly,  quarterly) 


Procedures: 

• Designate  a staff  member  to  coordinate  the  collection  of  this  statistic. 

• Direct  the  designated  staff  member  to  contact  library  staff  members  who  are  in  charge  of  digital 
collection  projects.  Ask  all  library  staff  members  involved  in  any  digital  collection  projects  as  part  of 
their  official  responsibilities  to  fill  out  the  worksheet  in  Appendix  C,  Figure  C.8,  for  the  reporting  period. 
Ask  them  to  estimate  how  much  of  their  time  was  spent  on  planning,  implementing,  and  managing 
digital  collection  projects.  This  information  will  be  entered  in  the  worksheet  as  FTE  (full-time 
equivalent).  A further  breakdown  of  activities  may  be  necessary  if  the  library  wants  to  have  more 
detailed  information  on  the  distribution  of  efforts. 

• Note  that  annual  salary  should  not  be  asked  of  the  staff  members  filling  out  the  worksheet  and  should  not 
include  fringe  benefits.  When  all  the  worksheets  are  collected,  the  salary  information  will  be  obtained 
from  the  library  accounting  or  personnel  department  Direct  staff  cost  will  then  be  calculated. 

• The  personnel  cost  should  also  include  wages  paid  to  non-salaried  staff,  including  student  and  other 
hourly  workers. 

• Cost  of  equipment  should  be  amortized.  For  example,  if  a $3,000  scanner  was  purchased  at  the 
beginning  of  the  reporting  year  and  has  a depreciation  period  of  three  years,  register  $ 1 ,000  as  the 
equipment  cost  Costs  of  software  should  be  reflected  in  full  amounts  based  on  the  time  of  the  purchase. 

• If  a subcontracting  period  is  different  from  the  reporting  period,  prorate  the  amount  for  the  reporting 
period.  If  the  payment  is  based  on  percent  to  completion,  include  only  the  amount  that  belongs  to  the 
reporting  period. 

• Use  the  sample  worksheet  in  Appendix  C,  Figure  C.9,  to  calculate  the  total  cost. 
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Sample  Digital  Collection  Cost 
Report  Form  - Personnel  - Figure  C.8  (for  D3) 


Reporting  Period:. 
Name  of  Library:_ 
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Sample  Digital  Collection  Cost 
Report  Form  - Figure  C.9  (for  D3) 

Reporting  Period: 

Name  of  Library: 
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Preface 


This  manual  provides  definitions,  data  collection 
procedures,  and  discusses  related  issues  pertaining  to  in- 
terpreting and  using  the  recommended  statistics  and  mea- 
sures. The  definitions  and  procedures  were  derived  from 
a month  of  field-testing  at  more  than  a dozen  ARL  librar- 
ies. The  statistics  and  performance  measures  represent  a 
minimum  set  of  data  that  need  to  be  collected  continually 
and  used.  Individual  libraries  will  need  to  develop  local 
procedures  to  support  data  collection  activities  within  the 
guidelines  of  this  manual.  However,  readers  need  to  rec- 
ognize that  the  statistics  and  measures  will  be  refined  and 
extended  continually  in  the  future.  PowerPoint  instruc- 
tional modules  to  accompany  this  manual  are  available 
from  ARL. 
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Introduction 


There  is  a critical  need  for  academic  and  research  libraries  to  develop  new 
statistics  and  measures  to  describe  network  services  and  resources.  This  manual  is 
one  product  of  a larger  New  Measures  Initiative  by  the  Association  of  Research 
Libraries  (ARL)  <http://www.arl.org/stats/newmeas/newmeas.html>.  The  E- 
Metrics  project  that  developed  this  manual  is  funded  by  a group  of  24  ARL  mem- 
ber libraries. 

Based  on  a substantial  field-testing  process  (described  in  detail  in  the  E- 
Metrics  Phase  II  report  <http://wnvw.arl.org/stats/newmeas/emetrics/>),  the  study 
team  recommends  a number  of  network  statistics  and  performance  measures  that 
provide  indicators  of  library  networked  services  and  resources. 

For  a host  of  reasons  not  explored  in  this  manual,  the  final  set  of  statistics  and 
measures  were  selected  among  others  that  were  considered  but  not  discussed  in  detail  herein. 
The  selection  of  certain  measures  over  others  is  not  meant  to  imply  that  those  not  selected 
are  flawed  or  have  no  beneficial  use.  See  Appendix  A for  a table  containing  three  lists  of 
candidate  measures  considered  during  the  E -Metrics  project,  and  refer  to  Part  2 of  the  E- 
Metrics  Phase  II  Report  for  a discussion  of  the  evolution  of  measures. 

These  statistics  and  measures  will  provide  research  libraries  with  an  impor- 
tant and  useful  set  of  tools  to  describe  and  assess  network  resources  and  services. 
The  manual  also  provides  libraries  with  guidance  regarding  the  use  to  which  the 
network  statistics  and  measures  can  be  put. 

The  manual  offered  here  has  a number  of  specific  goals  and  objectives.  Its 
primary  goal  is  to  provide  a beginning  approach  for  research  libraries  to  better 
describe  the  use  and  users  of  their  networked  services.  A secondary  goal  is  to 
increase  the  visibility  and  importance  of  developing  such  statistics  and  measures. 
Specific  objectives  of  the  manual  are  to: 

□ Identify  selected  key  statistics  and  measures  that  can  describe  use  and 
users  of  electronic  and  networked  services; 

□ Standardize  procedures  and  definitions  to  collect  these  statistics  and 
measures;  and 

□ I ncrease  awareness  of  selected  issues  related  to  collecting,  analyzing,  and 
reporting  the  data  to  produce  these  statistics  and  measures. 

The  statistics  and  measures  offered  here  will  need  to  be  continually  developed, 
expanded,  refined,  and  possibly  eliminated  over  time. 

A key  component  of  the  project  has  been  to  work  with  vendors  and  other 
organizations  regarding  the  collection,  manipulation,  and  reporting  of  vendor- 
supplied  online  database  data.  Many  of  the  statistics  described  here  resulted  from 
the  cooperative  efforts  among  these  vendors  and  other  national/international 
groups  interested  in  developing  such  statistics.  Such  efforts  should  be  continued. 

Given  the  rapidly  changing  technology  environment,  the  changing  milieu  of 
higher  education,  changing  organizational  structures  within  ARL  libraries,  and  the 
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complexity  of  measuring  such  networked  services,  it  is  almost  certain  that  the 
statistics  and  measures  proposed  in  this  manual  will  continue  to  evolve.  These 
measurement  tools,  however,  will  provide  research  librarians  with  important 
techniques  to  count,  describe,  and  report  networked  services  and  resources  in  their 
libraries. 
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and  Research  Library  Network  Statistics 
and  Perfonnance  Measures 


Network  Statistics 


Patron  Accessible  Electronic  Resources 

□ R1  Number  of  electronic  full-text  journals  (p.  5) 

□ R2  Number  of  electronic  reference  sources  (p.  7) 

□ R3  Number  of  electronic  books  (p.  8) 

Use  of  Networked  Resources  and  Services 


□ Ul  Number  of  electronic  reference  transactions  (p.  11) 

□ U2  Number  of  logins  (sessions)  to  electronic  databases  (p.  13) 

□ U3  Number  of  queries  (searches)  in  electronic  databases  (p.  14) 

□ U4  Items  requested  in  electronic  database  (p.  15) 

□ U5  Virtual  visits  to  library’s  website  and  catalog  (p.  16) 

Expenditures  for  Networked  Resources  and  Related  Infrastructure 

□ Cl  Cost  of  electronic  full-text  journals  (p.  20) 

□ C2  Cost  of  electronic  reference  sources  (p.  21) 

□ C3  Cost  of  electronic  books  (p.  22) 

□ C4  Library  expenditures  for  bibliographic  utilities,  networks,  and  consortia  (p.  24) 

□ C5  External  expenditures  for  bibliographic  utilities,  networks,  and  consortia  (p.  25) 

Library  Digitization  Activities 

□ Dl  Size  of  library  digital  collection  (p.  27) 

□ D2  Use  of  library  digital  collection  (p.  29) 

□ D3  Cost  of  digital  collection  construction  and  management  (p.  30) 

Performance  Measures 

Performance  Measures 

□ Pi  Percentage  of  electronic  reference  transactions  of  total  reference  (p.  33) 

□ P2  Percentage  of  virtual  library  visits  of  all  library  visits  (p.  34) 

□ P3  Percentage  of  electronic  books  to  all  monographs  (p.  35) 
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Criteria  for  Performance  Statistics  and  Measures 


The  data  collection  procedures  for  the  statistics  and  performance  measures 
are  defined  and  described  according  to  the  following  criteria: 

Definition:  Describes  each  statistic  or  performance  measure. 

Rationale:  Discusses  why  the  suggested  statistic  or  performance  measure  is 
needed  and/or  how  it  can  be  useful  to  describe  electronic  resources  and 
services. 

implementation:  Provides  instructions  for  implementing  the 

identified  statistic  or  performance  measure,  categorized  by  collector, 
frequency,  procedures,  and  special  considerations,  if  any. 

Collected  by:  Identifies  who  is  responsible  for  collecting  data; 

local  refers  to  the  individual  library  and  vendors  refers  to  the  content 
providers  with  whom  the  library  has  contracted  to  provide  electronic 
resources. 

Frequency:  Identifies  how  often  the  statistic/measure  needs  to  be 

collected. 

Procedures:  Outlines  the  manner  in  which  the  data  for  a statistic 

or  performance  measure  may  be  collected.  Also  includes  recommendations 
for  forms. 

Special  considerations:  Identifies  special  factors  that  need  to  be 
considered  during  data  collection  or  interpretation. 

Related  issues:  Discusses  issues  that  go  beyond  the  suggested 

data  collection  procedures,  such  as  the  availability  of  complementary 
statistics,  ways  in  which  statistics  can  be  combined  with  other  statistics, 
and  other  possible  approaches  to  data  collection. 
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and  Measures 


Statistics  Related  to  Patron  Accessible  Resources 


The  statistics  developed  for  patron  accessible  resources  account  for 
networked  resources  and  services.  The  current  ARL  membership 
criteria  index  lacks  separate  measures  for  electronic  and  networked 
monographs,  serials,  and  bibliographic  utilities.  Though  these  electronic  and 
networked  resources  may  limit  the  amount  of  print  materials  acquired  and  may  cost 
more  than  their  print  counterparts,  they  do  constitute  more  widely  available 
resources. 

In  the  electronic  and  networked  realm,  the  more  a library  has,  the  more 
materials  are  provided  to  customers  anytime  and  anywhere.  Although  local  needs 
and  available  resource  allocations  may  differ  from  library  to  library,  the  resource 
statistics  allow  academic  research  libraries  to  see  and  to  demonstrate  to  others  the 
changing  nature  of  library  collections  over  the  years.  In  turn,  the  libraries  are 
expected  to  use  them  to  make  decisions  about  resource  allocations  (budget,  staff, 
time,  etc.)  and  to  undertake  strategic  planning  accordingly.  Furthermore,  the 
picture  of  available  resources  provides  libraries  with  an  opportunity  to  offer  valued 
services.  However,  because  the  evolving  nature  of  these  statistics  will  rely  heavily 
on  technological  enhancements,  all  libraries  are  encouraged  to  use  extra  caution 
while  serving  their  institutional  goals,  missions,  and  visions. 


□ Rl  Number  of  electronic  full-text 
journals 

□ R2  Number  of  electronic  reference 
sources 

□ R3  Number  of  electronic  books 


Patron  Accessible 
Electronic 
Resources 


Number  of  electronic  full-text 
Journals 


Definition:  Number  of  electronic  full-text  journal  subscriptions  that 
the  library  provides  to  users  either  through  an  individual  institutional  licensing 
contract  with  the  provider  of  journals  or  through  other  arrangements  (e.g.,  regional 
or  state  consortium)  for  which  the  library  pays  a reduced  or  no  fee  for  access. 

The  full-text  journals  should  provide  both  search  and  browse  capabilities  by 
title  and  issue.  This  is  different  from  journal  article  databases,  such  as  Expanded 
Academic  ASAP  in  INFOTRAC,  that  do  not  provide  browsing  capability. 
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This  includes  electronic  full-text  journals  offered  by  established  scholarly 
journal  publishing  houses  (e.g.,  Elsevier’s  ScienceDirect  and  Academic  Press’s 
IDEAL),  scholarly  societies  (e.g.,  American  Chemical  Society  journals  and  Ameri- 
can Institute  of  Physics  Online),  and  services  which  aggregate  content  from 
smaller  publishers  or  from  those  publishers  that  prefer  to  use  an  external  delivery 
platform  (Highwire,  OCLC  ECO,  and  EbscoOnline).  This  should  exclude  gen- 
eral-purpose periodicals  such  as  magazines  and  newspapers. 

Rationale:  Electronic  access  has  expanded  dramatically  to  provide  a 

range  of  useful  resources  for  library  users.  This  statistic  helps  document  the 
degree  of  expansion  of  electronic  resource  availability  in  the  individual  library  and 
can  be  used  to  justify  continuation  and  enhancement  of  these  services. 

Research  libraries  act  increasingly  as  gateways  to  a vast  array  of  external 
information.  This  measure  specifically  addresses  the  extensiveness  of  scholarly 
content  a library  provides  to  its  user  community.  In  many  cases,  electronic  access 
enables  the  library  to  offer  larger  selections  of  journals  than  it  could  provide  in 
paper  format.  This  statistic  can  also  be  used  for  library  promotion  and  internal 
and  external  reporting.  Particularly,  this  statistic  aims  at  showing  the  changing 
nature  of  traditional  scholarly  resources  with  improved  and  better  access  anytime 
and  anywhere. 

Implementation 

□ Collected  by:  Local  and  vendors 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent 
(e.g.,  monthly,  quarterly) 

□ Procedures:  It  is  impossible  to  obtain  the  complete  list  of  electronic  full-text 
journals  from  a single  source.  Possible  sources  for  the  information  include 
library  catalog  records  (those  records  that  point  to  web  addresses),  library  web 
pages  that  list  the  journal  titles,  the  internal  electronic  resource  management 
database,  and  vendor  records  (websites  and  contract  documentation). 

1.  Create  a master  list  of  full-text  electronic  journals  from  all  the  sources 
available.  Use  a spreadsheet  or  database  program  to  organize  and  maintain 
the  list. 

2.  Remove  titles  that  do  not  meet  the  above-mentioned  definition  but  keep 
duplicate  titles  resulting  from  multiple  subscriptions  (e.g.,  main  library 
and  medical  or  law  libraries). 

3.  Record  the  counts  and  be  sure  they  are  updated  regularly.  The  library 
should  also  update  the  count  information  on  the  library  website  and/or  in 
marketing  brochures  on  a regular  basis. 

Special  considerations:  It  is  time-consuming  to  establish  procedures 
to  collect  this  statistic  for  the  first  time.  However,  once  that  is  done,  it  will  be 
relatively  easy  to  update  the  information.  (This  applies  to  other  statistics  and 
measures  included  in  the  manual  as  well.) 

Include  journal  titles  that  come  with  print  subscriptions  or  print  plus  online 
subscriptions  since  the  focus  of  the  statistic  has  to  do  with  how  many  scholarly 
electronic  journal  titles  users  can  access.  Do  not  include  free  government  publica- 
tions and  free  electronic  journals  to  which  the  library  provides  links. 
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Free  government  publications  and  free  electronic  journals  are  a valuable 
resource  for  many  libraries.  How  to  collect  statistics  relating  to  these  resources  will 
be  addressed  in  the  future. 


Number  of  electronic  reference  sources 


Definition:  Number  of  electronic  reference  sources  and  aggregation 
services  that  the  library  provides  to  users  either  through  an  individual  licensing 
contract  with  the  content  providers  or  through  other  arrangements  (e.g.,  regional 
or  state  consortium)  for  which  the  library  pays  a reduced  or  no  fee  for  access. 

This  includes  citation  indexes  and  abstracts;  full-text  reference  sources  (e.g. 
encyclopedias,  almanacs,  biographical  and  statistical  sources,  and  other  quick  fact- 
finding sources);  full-text  journal  and  periodical  article  collection  services  (e.g., 
EBSCOhost,  ProQuest,  Academic  Universe,  and  INFOTRAC  OneFile);  dissertation 
and  conference  proceedings  databases;  and  general-purpose  magazines  and  newspa- 
pers. Licensed  electronic  resources  also  include  those  databases  that  institutions 
mount  locally 

Rationale:  Networking  technology  in  libraries  has  improved  and 
increased  dramatically  user  access  to  a range  of  useful  reference  resources.  This 
statistic  documents  the  degree  of  expansion  of  electronic  resource  availability  and 
can  be  used  to  justify  continuation  and  enhancement  of  these  services.  In  the 
1990s,  because  of  the  increasing  popularity  of  the  Internet,  the  ways  reference 
interviews  were  held  and  reference  sources  were  used  changed.  Today,  users  have 
electronic  formats  as  well  as  traditional  reference  sources  to  provide  answers  to 
their  reference  questions. 

Research  libraries  traditionally  act  as  gateways  to  a vast  array  of  external 
information.  This  measure  deals  with  the  extensiveness  of  scholarly  content  the 
library  provides  to  the  user  community  and  the  availability  of  reference  sources  on 
an  anytime/anywhere  basis.  In  many  cases,  electronic  access  enables  the  library  to 
offer  more  resources  than  it  could  in  paper  format.  This  statistic  can  also  be  used 
for  library  promotion  and  internal  and  external  reporting.  Specifically,  this  statistic 
aims  at  showing  the  changing  nature  of  traditional  scholarly  resources  with  im- 
proved access. 

Implementation 

□ Collected  by:  Local  and  vendors 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent 
(e.g.,  monthly,  quarterly) 

□ Procedures:  As  in  the  case  of  the  number  of  electronic  full-text  journals,  it  is 
impossible  to  obtain  the  complete  list  of  databases  from  a single  source.  Pos- 
sible sources  for  the  information  include  library  catalog  records  (those  records 
that  point  to  web  addresses),  library  web  pages  that  list  the  database  titles,  the 
internal  electronic  resource  management  database,  and  vendor  records  (websites 
and  contract  documentation). 
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1.  Create  a master  list  of  electronic  databases  from  all  the  sources  available. 
Use  a spreadsheet  or  database  program  to  organize  and  maintain  the  list. 

2.  Remove  titles  that  do  not  meet  the  above-mentioned  definition  but  keep 
duplicate  titles  resulting  from  multiple  subscriptions  (e.g.,  main  library  and 
medical  or  law  libraries). 

3.  Record  the  counts  and  be  sure  they  are  updated  regularly.  The  library 
should  also  update  the  count  information  on  the  library  website  and/or  in 
marketing  brochures  on  a regular  basis. 

Special  considerations:  The  unit  of  measurement  here  is  the  data- 
base not  the  whole  service  provided  by  a vendor.  For  example,  if  the  library 
subscribes  to  OVID  and  the  company  provides  five  databases  (ABI/Inform,  Books 
in  Print,  CINAHL,  INSPEC,  and  PsycINFO),  then  the  count  is  5,  not  1.  By  the 
same  token,  if  the  library  subscribes  to  three  database  packages  (Academic  Uni- 
verse, Congressional  Universe,  and  Statistical  Universe)  from  Lexis-Nexis,  the 
count  is  3. 

This  count  should  not  include  freely  available  databases  to  which  the  library 
provides  links  or  library-created  finding  aids. 

Freely  available  databases  and  library-created  finding  aids  are  a valuable 
resource  for  many  libraries.  How  to  collect  statistics  relating  to  these  resources  will 
be  addressed  in  the  future. 

Number  of  Electronic  Books 

Definition:  Number  of  electronic  full-text  monographs  that  the 
library  offers  to  its  users  either  through  an  individual  licensing  contract  with  the 
content  providers  or  through  other  arrangements  (e.g.,  regional  or  state  consor- 
tium) where  the  library  pays  a reduced  or  no  fee  for  access. 

This  includes  electronic  books  purchased  through  vendors,  such  as  netLibrary 
and  Books24x7,  and  electronic  books  that  come  as  part  of  aggregate  services.  It 
excludes  internally  digitized  electronic  books,  electronic  theses  and  dissertations, 
digitally  created  archival  collections  (e.g.,  Early  English  Books  Online),  and  other 
special  collections.  This  also  excludes  publicly  available  electronic  books  to  which 
the  library  provides  web  links.  It  does  not  include  machine-readable  books  distrib- 
uted on  CD-ROM,  or  accompanied  by  print  books. 

Rationale:  Networking  technology  in  libraries  has  improved  and 
increased  dramatically  user  access  to  the  electronic  counterparts  of  some  traditional 
sources.  This  statistic  documents  the  degree  of  expansion  of  e-books.  In  the  mid 
90s,  networking  and  resource  sharing  technologies  provided  libraries  with  print 
books  and  e-books  that  were  made  available  through  a library’s  networks. 

Because  the  evolving  nature  of  this  statistic  will  heavily  depend  on  techno- 
logical enhancements,  all  libraries  are  encouraged  to  use  extra  caution  while 
pursuing  their  institutional  goals,  missions,  and  visions.  Moreover,  the  definition 
of  e-books  is  still  evolving.  This  statistic  is  an  early  attempt  to  keep  track  of  this 
type  of  resource  as  it  becomes  more  widely  available. 
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Implementation 

□ Collected  by:  Local  and  vendors 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent 
(e.g.,  monthly,  quarterly) 

□ Procedures: 

1.  For  each  electronic  book  collection,  get  the  electronic  title  counts  from 
either  the  providers  or  catalog  records.  Unlike  electronic  full-text  journals 
and  reference  databases,  it  is  not  necessary  to  list  the  titles  for  each  elec- 
tronic book  collection. 

2.  Count  any  duplicate  titles  resulting  from  multiple  subscriptions  (e.g.,  main 
library  and  medical  or  law  libraries). 

3.  Record  the  counts  and  be  sure  they  are  updated  regularly.  The  library 
should  also  update  the  count  information  on  the  library  website  and/or  in 
marketing  brochures  on  a regular  basis. 

Special  considerations:  Do  not  include  book  collections  that  are  a 
part  of  aggregate  services  and  function  more  as  a reference  collection  (e.g.,  MD 
Consult  reference  books,  ProQuest's  Early  English  Books  Online,  and 
books@OVID).  They  should  be  reported  in  the  electronic  reference  databases. 

Do  not  include  freely  available  electronic  books  such  as  titles  available  from 
the  National  Academy  Press. 

Related  issues:  Electronic  books,  still  evolving  in  terms  of  technol- 
ogy and  adoption  for  use,  present  a number  of  issues  in  terms  of  definition  and 
measurement,  such  as  “location,”  accessibility  (metadata  and  access  points),  and  use 
versus  circulation  (e.g.,  is  online  use  for  20  minutes  a circulation,  as  it  would  be 
with  reserve  materials,  or  does  a circulation  of  electronic  books  require  a minimum 
period  of  use,  such  as  24  hours?). 

□ What  about  reference  book  collections  provided  by  vendors?  Should  they 
be  treated  as  electronic  books,  for  example,  or  should  they  be  treated  as  a 
database,  on  the  grounds  that  they  are  used  as  databases? 

Count  only  those  books  that  a user  can  check  out,  as  they  would  traditional 
books.  Unlike  traditional  books  that  the  library  purchases  and  owns,  electronic 
books  can  be  subscribed  to  for  an  ongoing  fee.  In  this  case,  the  library  accounting 
system  may  treat  these  as  serials  rather  than  books  because  of  the  type  of  payment. 
It  is  relatively  easy  to  keep  track  of  the  number  of  electronic  books  right  now  since 
most  libraries  deal  with  only  a handful  of  e-book  vendors,  such  as  netLibrary  and 
Books24x7.  But  in  the  future,  it  will  become  increasingly  difficult  to  do  this  as  the 
sources  of  electronic  books  proliferate. 

Finally,  some  provisions  of  contractual  agreements  between  libraries  and 
vendors  may  limit  the  level  of  use  of  e-books.  These  issues  need  be  addressed  in 
future  research. 
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Dse  of  Electronic  Networked  Resources  and  Services 


High  use  of  a library  resource  or  service  implies  a collection  development 
program  that  is  working  to  create  access  to  the  resources  customers 
need.  Use  and  the  need  can  also  identify  resources  and  services  that  are  seen  as 
particularly  valuable  in  the  education  and  research  enterprise  and  should  be  ex- 
panded, or  perhaps  resources  and  services  that  should  be  discontinued  due  to  lack 
of  use  and  interest.  Whether  provided  by  vendors  or  collected  institutionally,  usage 
statistics  can  help  a library  administrator  make  decisions  and  plan  for  the  future  in 
order  to  meet  not  only  users’  expectations  and  needs  but  also  institutional  goals. 

The  reported  data  can  also  provide  other  information  as  to  where  and  when  people 
use  the  library’s  materials  and  how  well  the  library  serves  its  target  audience  and 
anticipates  their  potential  needs. 


The  cost  of  providing  access  to  networked  resources  and  services  can  be  more 
expensive  than  that  of  traditional  counterparts.  Depending  heavily  on  earlier 
ICOLC  guidelines,  the  E-Metrics  use  measures  put  this  in  the  perspective  of  the 
changing  academic  research  library  environment.  The  purpose  of  the  use  measures 
is  to  provide  statistics  relating  to  the  use  of  networked  services  and  resources. 
Therefore,  it  is  expected  that  library  administrators  can  reconsider  some  resource 
allocation  issues  as  the  number  of  resources  and  services  tend  to  increase  while 
people  are  provided  greater  access.  Please  note  that,  as  with  most  of  the  statistics 
in  this  study,  statistics  related  to  the  use  of  library  resources  and  services  should  be 
revisited  and  perhaps  modified  as  the  technology  advances. 
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□ Ul  Number  of  electronic  reference 
transactions 

□ U2  Number  of  logins  (sessions)  to 
electronic  databases 

□ U3  Number  of  queries  (searches)  in 
electronic  databases 

□ U4  Items  requested  in  electronic  database 

□ U5  Virtual  visits  to  library’s  website  and 
catalog 


Use  of  Networked  Resources 
and  Services 


Number  of  electronic  reference 

TRANSACTIONS 


•3 


Definition:  Number  of  electronic  reference  transactions  conducted 

via  email,  a library’s  website,  or  other  network  communications  mechanisms 
designed  to  support  electronic  reference.  An  electronic  reference  transaction  must 
include  a question  either  received  electronically  (e.g.,  via  e-mail,  WWW  form,  etc.) 
or  responded  to  electronically.  Those  transactions  that  are  both  received  and 
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responded  to  electronically  are  counted  as  one  transaction.  This  count  excludes 
phone  and  fax  traffic  unless  either  the  question  or  answer  transaction  occurs  via  the 
described  manner.  It  includes  the  counts  accrued  from  participation  in  any  local 
and  national  projects,  such  as  DigiRef  and  the  Library  of  Congress's  CDRS  (Col- 
laborative Digital  Reference  Service). 

A reference  transaction  is  an  information  contact,  which  involves  the  knowl- 
edge, use,  recommendations,  interpretation,  or  instruction  in  the  use  of  one  or 
more  information  sources  by  a member  of  the  library  staff. 

Rationale:  Libraries  are  making  more  of  their  services  available 
electronically  and  are  interested  in  tracking  the  development  of  a new  and  emerg- 
ing library  service.  There  is  a need  to  better  document  this  transition  to  facilitate 
and  improve  resource  allocation  activities.  This  statistic  represents  reference 
activities  conducted  electronically  in  the  library.  It  is  an  attempt  to  measure 
reference  transactions  through  new  electronic  tools. 

Implementation 

□ Collected  by:  Local 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent 
(monthly,  quarterly).  This  statistic  can  be  collected  in  the  same  manner  as  the 
library  gathers  other  reference  transactions  data. 

□ Procedures: 

1.  Select  a typical  week  (or  month)  to  run  a sample  study.  Be  sure  to  vary  the 
specific  week  (or  month)  chosen  over  the  course  of  a year  or  from  year  to 
year  to  account  for  seasonal  fluctuations. 

2.  Key  tasks  include  distributing  a daily  tally  sheet,  collecting  the  daily  tally 
sheet,  adding  each  day’s  totals  to  a weekly  figure,  and  being  available  to 
respond  to  data  collection  problems  should  they  occur. 

3.  Transactions  may  be  via  e-mail,  a form  on  a web  page,  etc.  Electronic 
reference  transactions  may  involve  more  than  reference  desk  staff  (e.g.,  web 
master,  various  reference  personnel,  library  director,  volunteers,  etc.). 
Establish  an  administrative  procedure  to  report  electronic  reference  trans- 
action counts  to  a designated  staff  person,  no  matter  who  receives  the 
questions  or  answers  the  reference  requests. 

4.  Disseminate  the  new  procedure  and  rationale.  Several  notices  throughout 
the  year  may  be  necessary. 

5.  Report  an  electronic  reference  transaction  as  you  would  a face-to-face 
reference  transaction.  Thus,  one  e-mail  request  may  contain  several 
reference  questions  taking  varying  times  to  complete.  For  example,  one  e- 
mail  request  could  contain  two  relatively  short  reference  questions  and  one 
reference  question  that  took  10-15  minutes  to  answer.  Count  the  number 
of  requests,  not  the  number  of  questions.  Thus,  in  the  example  you  would 
report  one  (1)  as  the  number  of  electronic  reference  transactions  even 
though  there  were  three  questions.  Report  counts  using  pre-established 
local  library  reporting  periods  (weekly,  monthly,  etc.). 

6.  Indicate  and  describe  any  additional  methods  used  outside  of  this  defini- 
tion and  guidelines. 
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Special  considerations:  Unless  the  library  uses  electronic  reference 
management  software  to  collect  and  report  transaction  data,  it  is  difficult  to  keep 
track  of  a complete  reference  transaction  cycle  (query  and  response)  because  of 
time-delays  and  the  involvement  of  several  parties. 


As  stated  in  the  definition,  the  statistic  includes  the  number  of  service 
transactions  provided  to  patrons  outside  the  university  or  the  parent  institution 
that  the  library  serves,  through  regional  or  national  cooperative  efforts  and  through 
library  policies. 

Related  issues:  Reference  services  are  undergoing  rapid  changes. 
Libraries  are  experimenting  with  different  modes  of  electronic  reference.  One 
could  say  that  simple  email  transactions  that  are  prominently  mentioned  in  the 
procedures  are  not  much  different  from  traditional  reference  services.  How  can  a 
library  measure  quality  in  providing  different  types  of  electronic  reference  services 
such  as  live-chat  with  text/voice/video?  Will  this  measure  help  the  library  deter- 
mine user  demand  and  thereby  plan  for  resource  allocation?  To  answer  these 
questions,  libraries  need  to  collect  more  detailed  information  such  as  length  of  time 
taken  to  answer  questions,  types  of  questions  by  types  of  transactions,  and  so  on. 
Also,  this  statistic  is  likely  to  produce  some  useful  figures  and  trends  regarding 
staff  support  and  allocation  in  reference  activities. 


Number  of  logins  (sessions)  to  electronic 

DATABASES 


Definition:  Number  of  user  initiated  sessions  in  licensed  electronic 

resources.  A session  or  login  is  one  cycle  of  user  activities  that  typically  starts 
when  a user  connects  to  a database  and  ends  with  explicit  termination  of  activities 
(by  leaving  the  database  through  logout  or  exit)  or  implicit  termination  (time  out 
due  to  user  inactivity).  Licensed  electronic  resources  also  include  those  databases 
that  institutions  mount  locally. 

Rationale:  One  purpose  of  having  a networked  environment  is  to 
promote  connectedness  and  accessibility  to  a variety  of  information  resources, 
hence  the  need  for  this  measure.  Also,  the  gradual  shift  in  the  materials  expendi- 
tures from  traditional  print-based  resources  to  electronic  databases  can  be  under- 
stood with  the  measure.  This  measure  will  produce  a count  of  how  often  specific 
databases  are  used  and  complement  traditional  physical  attendance  counts. 

Implementation:  Until  there  is  a standardized  report  generation  system 
that  captures  different  statistics  from  different  content  providers,  we  recommend 
that  each  library  develop  an  in-house  spreadsheet  or  database  to  capture  monthly 
usage  statistics  of  licensed  databases.  At  least  on  the  database  title  level,  usage 
statistics  should  be  collected  from  vendors,  entered  into  the  in-house  databank,  and 
maintained  for  reporting  and  analysis. 

□ Collected  by:  Vendor 

□ Frequency:  Monthly,  but  can  be  reported  quarterly  or  annually 

□ Procedures: 
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1.  Process  monthly  usage  statistics  from  vendors  and  copy  or  import  the 
number  of  attempted  sessions  in  each  database  (in  each  journal  collection 
for  full-text  journals)  to  an  in-house  spreadsheet  or  database  file. 

2.  Calculate  the  total  sessions  for  a given  month  by  adding  the  number  of 
sessions  from  each  database  or  journal  collection. 

Special  considerations:  Not  all  vendors  report  this  statistic.  There- 

fore, it  will  be  necessary  to  qualify  the  statistic  with  a sentence  such  as  this:  “We 
have  150,000  logins  recorded  from  120  databases  out  of  200  subscribing.  We 
cannot  report  this  statistic  for  the  remaining  80  databases  because  the  vendor  does 
not  supply  login  (session)  information  to  customers.” 

Related  issues:  When  analyzing  the  login  counts,  it  might  be  impor- 
tant to  explain  any  increases  or  decreases  in  the  figures.  Specify,  for  example, 
whether  the  increase  comes  from  (1)  the  addition  of  new  databases,  (2)  databases 
which  did  not  report  the  statistic  in  the  past  but  have  now  begun  reporting,  (3) 
increased  demand,  and/or  (4)  an  increase  in  the  number  of  simultaneous  users. 

Problems  with  the  comparability  of  login  counts  from  different  vendors  is  a 
serious  threat  to  the  utility  of  the  combined  count.  Content  providers  use  different 
time-out  thresholds  (ranging  from  7 to  30  minutes  on  average).  Also,  because  of 
the  IP-based  authentication,  several  sessions  conducted  at  the  same  public  work- 
station can  be  counted  as  a single  login.  Alternatively,  libraries  can  collect  at- 
tempted logins  to  various  licensed  databases  by  making  users  go  through  a central 
gateway  (which  counts  all  attempted  logins).  This  will  ensure  that  one  login 
attempt  to  a database  is  the  same  as  a login  to  other  databases.  However,  what  this 
data  collection  method  misses  is  user  logins  that  go  directly  to  content  provider 
sites.  It  is  unclear  how  many  user  logins  fall  into  this  category,  but  the  phenom- 
enon certainly  results  in  a substantial  undercount  of  user  logins. 

While  the  gross  login  figure  is  useful,  it  is  useful  only  for  trend  plotting  and 
gross  justification  of  electronic  resources.  Within  the  library,  the  usage  measures 
of  licensed  electronic  resources  have  many  users  and  uses.  Circulation  of  usage 
statistics  on  the  database  title  level  (or  in  an  extreme  case  on  the  journal  title  level) 
and  discussion  of  any  noticeable  changes  (or  lack  thereof)  need  to  occur  at  various 
levels  among  the  concerned  parties,  including  collection  development  personnel, 
web  master(s),  technical  services  staff,  and  so  on. 


Number  of  queries  (searches)  in  Electronic 
Databases 


Definition:  Number  of  user  initiated  queries  (searches)  in  licensed 
electronic  resources.  A search  is  intended  to  represent  a unique  intellectual  in- 
quiry. Typically,  a search  is  recorded  each  time  a search  request  is  sent/submitted 
to  the  server. 

Rationale:  This  statistic  provides  libraries  with  an  indication  of  the 
databases  that  are  most  heavily  used,  areas  of  user  interest,  database  popularity,  and 
a level  of  usage  detail  that  goes  beyond  an  initial  session.  It  also  can  provide 
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important  information  for  billing  purposes,  as  some  vendors  charge  for  database 
usage  by  number  of  searches.  This  statistic  can  complement  Ul,  the  number  of 
electronic  reference  transactions,  as  more  user  requests  bypass  staff  mediations. 
Some  portion  of  this  statistic  is  also  analogous  to  in-library  use  of  reference 
sources. 

Implementation 

D Collected  by:  Vendor 

□ Frequency:  Monthly,  but  can  be  reported  quarterly  or  annually 

□ Procedures:  Until  there  is  a standardized  report  generation  system  that  captures 
different  statistics  from  different  content  providers,  we  recommend  that  each 
library  develop  an  in-house  spreadsheet  or  database  to  capture  monthly  usage 
statistics  of  licensed  databases.  Usage  statistics  need  to  be  collected  from 
vendors,  entered  into  the  in-house  databank,  and  maintained  for  reporting  and 
analysis. 

1.  Process  monthly  usage  statistics  from  vendors  and  copy  or  import  the 
number  of  attempted  searches  in  each  database  to  an  in-house  spreadsheet 
or  database  file. 

2.  Calculate  the  total  number  of  searches  for  a given  month  by  adding  the 
number  of  searches  from  each  database  or  journal  collection. 

Special  considerations:  Because  some  vendors  do  not  report  this 
statistic,  it  will  be  necessary  to  qualify  the  statistic  with  a sentence  such  as  this: 
“We  have  150,000  searches  recorded  from  120  databases  out  of  200  subscribing. 
The  other  80  do  not  provide  this  statistic.” 

Related  issues:  Different  assumptions  about  and  mechanisms  for 
collecting  search  counts  by  different  vendors  are  potential  threats  to  the  combined 
count. 


Items  requested  in  electronic  databases 


Definition:  Number  of  items  requested  in  all  of  the  library’s  licensed 
electronic  resources.  These  resources  may  include  journal  articles,  e-books,  refer- 
ence materials,  and  non-textual  resources  that  are  provided  to  the  library’s  users 
through  licensing  and  contractual  agreements.  The  user  requests  may  include 
viewing,  downloading,  emailing,  and  printing  to  the  extent  the  activity  can  be 
recorded  and  controlled  by  the  server  rather  than  browser. 

The  items  reported  depend  on  the  type  of  content.  Examples  include  cita- 
tions, abstracts,  tables  of  contents,  and  full-text  articles  (ASCII,  HTML,  PDF,  or 
PS). 

Rationale:  This  statistic  provides  a circulation  count  for  electronic 
contents  in  a way  analogous  to  the  traditional  circulation  of  books.  Given  the  fact 
that  libraries  do  not  have  good  measurements  of  in-house  materials  usage,  particu- 
larly serials  usage,  this  statistic  helps  libraries  understand  in-library  use  patterns 
that  were  heretofore  difficult  to  measure. 
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Implementation 

O Collected  by:  Vendor 

□ Frequency:  Monthly,  but  can  be  reported  quarterly  or  annually 

□ Procedures:  Until  there  is  a standardized  report  generation  system  that  captures 
different  statistics  from  different  content  providers,  we  recommend  that  each 
library  develop  an  in-house  spreadsheet  or  database  to  capture  monthly  usage 
statistics  of  licensed  databases.  Usage  statistics  should  be  collected  from  ven- 
dors, entered  into  the  in-house  databank,  and  maintained  for  reporting  and 
analysis. 

1.  Process  monthly  usage  statistics  from  vendors  and  copy  or  import  the 
number  of  items  selected  for  viewing,  downloading,  and  emailing  in  each 
database.  Count  the  number  and  type  of  items  users  selected:  abstracts, 
citations,  and  full-texts. 

2.  Calculate  the  total  number  of  items  for  a given  month  by  adding  the 
number  of  items  requested  from  each  database  or  journal  collection. 

Special  considerations:  Because  some  vendors  do  not  report  this 

statistic,  it  will  be  necessary  to  qualify  the  statistic  with  a sentence  such  as  this: 
“More  than  150,000  items  were  requested  from  120  databases  out  of  200  subscrib- 
ing. The  other  80  do  not  provide  this  statistic.  Among  the  requested  items, 
100,000  were  some  form  of  full-text  records.” 

Related  issues:  Different  vendors  apply  different  assumptions  and 

mechanisms  in  collecting  items  requested  counts.  This  lack  of  standardization 
makes  it  difficult  to  calculate  an  aggregate  count. 

We  do  not  have  good  measurement  of  in-house  materials  usage,  particularly 
journal  usage.  However,  electronic  journals  and  databases  allow  libraries  to  find 
out  how  often  materials  are  requested.  Having  in-house  usage  figures  is  important 
for  understanding  the  dynamics  of  usage  between  print  and  electronic  journals,  so 
that  we  can  ascertain  any  correlation  between  them. 


Virtual  visits  to  library's  website  and  catalog 


Definition:  This  is  defined  as  user  visits  to  the  library’s  website  or 
catalog  from  outside  the  physical  library  premises  regardless  of  the  number  of 
pages  or  elements  viewed.  If  a user  looks  at  16  pages  and  54  graphic  images  while 
at  a website,  that  user  registers  one  visit  on  the  web  server.  All  visits  to  the  website 
should  be  counted  regardless  of  repetition  by  one  user.  A visit  is  usually  deter- 
mined by  a user’s  IP  address,  which  can  be  misleading  due  to  Internet  Service 
Providers  (ISPs)  and  Firewalls  or  Proxy  Servers.  Thus,  this  measure  is  actually  an 
estimate  of  the  visits. 

Rationale:  Use  of  the  website  or  catalog  from  outside  the  library 
reflects  interest  in  library  services.  The  role  of  networked  services  is  to  expand  the 
reach  of  libraries  beyond  their  physical  boundaries.  This  statistic  helps  describe 
the  significance  of  networked  services  use  by  measuring  the  number  of  virtual 
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accesses.  This  will  also  give  an  opportunity  for  the  library  to  compare  the  demand 
placed  on  their  networked  resources  with  that  for  other  popular  information- 
oriented  websites  (such  as  Excite,  Lycos,  etc.). 

Implementation: 

□ Collected  by:  Local 

□ Frequency:  Reported  annually,  although  internal  reporting  will  be  more  frequent 
(e.g.,  weekly,  monthly,  and  quarterly). 

□ Procedures: 

1.  Identify  all  sources  of  virtual  visits  to  the  library.  This  may  involve  activi- 
ties that  take  place  on  more  than  one  web  server.  Some  of  the  web  servers 
may  be  owned  by  the  library  and  some  may  be  owned  or  maintained  by 
another  department  in  the  university,  an  Internet  Service  Provider  (ISP),  or 
other  library  vendors  (e.g.,  library  OPAC  provider). 

2.  Exclude  internal  use  within  the  premises  of  the  library  from  the  counts  for 
this  measure  when  possible.  Two  common  approaches  are  using  IP  ad- 
dresses or  some  form  of  authentication  tagged  to  each  transaction.  In 
terms  of  external  visits  to  the  library,  three  common  sources  are:  external 
access  to  the  library’s  web  page,  remote  logins  (sessions)  to  non-web-based 
library  databases,  and  remotely  accessible  library  OPAC. 

3.  Develop  strategies  for  collecting  the  data  from  each  of  these  sources  of 
virtual  visits.  Different  software  may  be  needed  to  measure  each  electronic 
source  of  virtual  visits.  In  some  cases,  the  library  may  calculate  the  virtual 
visits  using  one  or  more  log  analysis  software  packages.  In  other  cases,  the 
external  owner  of  the  web  server  or  service  (the  ISP)  must  provide  the  data. 
Discussions  may  need  to  be  held  with  these  service  providers  to  obtain  the 
needed  data.  In  still  other  cases,  custom  programs  may  have  to  be  devel- 
oped. 

4.  In  the  case  of  library  web  pages  housed  on  the  library  server,  identify, 
configure,  and  install  appropriate  log  analysis  software.  Determine  log 
analysis  software  definition  that  corresponds  to  the  virtual  visit  definition. 

Note:  Different  log  analysis  software  packages  may  count  virtual  visits  in 
different  ways,  so  the  count  obtained  will  by  necessity  be  an  estimate. 
Arrange  with  the  server  technical  staff  for  regular  (monthly)  reporting  of 
internal  visits  at  the  various  user  access  Internet  workstations,  external 
library  user  virtual  visits,  and  total  virtual  visits  (internal  visits  plus  exter- 
nal visits).  Run  the  log  analysis  software. 

5.  In  the  case  of  library  web  pages  housed  on  an  ISP’s  server,  identify  the  log 
analysis  software  the  ISP  uses.  Determine  the  definition  of  “visit”  used  by 
the  log  analysis  software  that  corresponds  to  the  virtual  visit  definition 
with  the  assistance  of  the  ISP.  Arrange  with  the  ISP  for  regular  (monthly) 
reporting  of  internal  library  visits  at  the  various  user  access  Internet 
workstations,  external  library  user  virtual  visits,  and  total  virtual  visits 
(internal  visits  plus  external  visits). 

6.  Where  virtual  visit  counts  include  the  aggregate  of  internal  and  external 
visits,  indicate  this  in  your  report. 
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Sp  ecial  considerations:  Count  all  visits  to  the  website  regardless  of 
repetition  by  one  user  as  long  as  each  visit  meets  the  criteria  for  this  statistic. 

After  one  user  connects  to  the  Internet,  several  users  could  conduct  multiple 
different  searches  in  the  electronic  service.  In  some  cases,  e.g.,  Internet-accessible 
OPAC  use  inside  the  library,  several  users,  one  after  the  other,  might  make  use  of 
the  same  established  connection.  In  most  systems,  a connection  is  cut  off  after  a 
specified  period  of  non-use,  thus  solving  part  of  the  problem.  The  best  existing 
method  of  collecting  virtual  visits  is  to  use  log  analysis  software.  The  log  analysis 
software  producers  may  define  virtual  visits  differently.  For  example,  does  a visit 
end  after  a time-out  period  of  30  minutes,  15  minutes,  or  some  other  time?  The 
recommended  time-out  period  is  30  minutes,  but  a local  library  may  have  to  accept 
the  available  log  analysis  software’s  definition  even  if  it  varies  from  the  above. 

Some  libraries  will  find  it  difficult  to  report  every  virtual  visit.  For  example, 
libraries  may  have  difficulty  counting  the  use  of  library  OPACS  because  their 
vendors  do  not  provide  this  information.  Make  a record  of  those  sources  of  virtual 
visits  not  counted.  Do  not  estimate  virtual  visits  for  which  data  are  not  available. 

Related  issues:  This  measurement  requires  a relatively  high  degree  of 

technical  skills  either  on  staff  or  available  from  the  library’s  website  host. 
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This  portion  of  the  statistics  is  based  on  the  ARL  Supplementary  Statistics 
Survey  (the  most  recent  survey  instruction  available  at  http;// 
www.arl.org/stats/arlstat/#sup).  In  collecting  the  statistics,  the  library  should  refer 
to  the  procedures  followed  and  the  amounts  reported  in  response  to  the  ARL 
Supplementary  Statistics  Survey. 

These  statistics  were  developed  by  ARL  to  determine  expenditure  patterns  on 
electronic  and  networked  resources  and  the  effect  of  new  types  of  library  resources 
and  services,  those  delivered  both  individually  and  collectively  with  other  institu- 
tions, on  library  expenditures.  These  measures  are  expected  to  help  ARL  libraries 
justify  their  growing  budgets  due  to  the  great  expense  of  electronic  and  networked 
services.  These  measures  can  help  answer  such  questions  as:  How  much  are 
research  libraries  spending  for  electronic  resources  collectively  and  how  much  on 
average?  How  do  expenditures  for  electronic  resources  compare  across  several 
research  libraries? 


'-D 


We  have  not  included  the  cost  of  the  technical  staff  and  their  training,  the 
networking  and  equipment  to  provide  access  to  the  electronic  resources  as  well  as 
the  time  of  all  the  staff  involved.  This  will  have  to  be  addressed  in  the  future. 

General  Introduction  to  C1-C3 


& 


The  report  should  include  expenditures  for  electronic  indexes  and  refer- 
ence tools,  electronic  full-text  periodical  collections  and  electronic 
journal  back-files,  and  online  searches  of  remote  databases  - whether  accessed 
remotely  or  installed  locally  from  CD-ROM,  magnetic  tapes,  etc.  The  report 
should  also  include  expenditures  for  materials  purchased  jointly  with  other  institu- 
tions if  such  expenditures  can  be  separated  from  other  charges  for  joint  services, 
fees  paid  to  bibliographic  utilities  if  the  portion  paid  for  computer  files  and  search 
services  can  be  separately  counted,  and  equipment  costs  when  they  are  inseparably 
bundled  into  the  price  of  the  information  product. 

Expenditures  for  bibliographic  utilities,  networks,  and  consortia  that  are 
unrelated  to  end-user  database  access  should  be  reported  in  C4,  not  in  Cl  through 

C3. 


rg) 


19 


; 398 

BEST  COPY  AVAILABLE 


□ Cl  Cost  of  electronic  full-text  journals 

□ C2  Cost  of  electronic  reference  sources 

□ C3  Cost  of  electronic  books 

□ C4  Library  expenditures  for  bibliographic  utilities, 
networks,  and  consortia 

□ C5  External  expenditures  for  bibliographic  utilities, 
networks,  and  consortia 


Cost  of  electronic  full-text  journals 

Definition:  Expenditures  for  electronic  full-text  journal  subscriptions 
that  the  library  provides  to  its  users.  Include  both  initial  purchase  cost,  member- 
ship fees  (such  as  JSTOR)  as  well  as  annual  access  and  service  fees  paid  directly  or 
through  consortia  arrangements. 

Rationale:  This  statistic,  cost  of  electronic  full-text  journals,  was 
developed  by  ARL  to  find  out  how  much  libraries  are  spending  on  electronic  full- 
text  journals  and  how  new  forms  of  electronic  journals  are  replacing  traditional 
journals  and  scholarly  publications.  It  also  indicates  the  extent  of  budget  alloca- 
tions for  electronic  resources.  Furthermore,  this  statistic  allows  libraries  to  calcu- 
late unit  costs  of  e-journals  after  collecting  Cl  and  Rl  statistics,  and  thus  aids 
libraries  in  deciding  how  effectively  they  are  serving  their  potential  and  intended 
audiences,  and  in  benchmarking  with  the  other  institutions. 

Implementation 

□ Collected  by:  Local 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent 
(e.g.,  monthly,  quarterly) 

□ Procedures:  For  the  definition  of  electronic  full-text  journals,  please  refer  to  the 
definition  of  Rl.  Current  library  accounting  systems  do  not  support  coding  of 
materials  expenditures  by  the  categories  used  in  the  manual.  Therefore,  it  may 
become  necessary  to  create  an  in-house  spreadsheet  or  database  file  to  keep 
track  of  cost  information  according  to  the  types  of  resources  (e-journals,  refer- 
ence databases,  and  e-books).  Preferably  a single  file  will  contain  contract 
information  (duration,  cost),  updated  title  counts,  and  reported  usage  statistics. 
Significant  coordination  is  required  for  setting  up  the  structure  of  the  file,  but  in 
the  long  run  may  streamline  many  aspects  of  the  management  of  electronic 
licensed  materials. 

1.  Gather  reports  and  invoices  related  to  electronic  databases  and  resources 
for  the  reporting  period.  These  documents  are  typically  handled  by  the 
library’s  accounting  office. 

2.  If  you  have  not  done  so,  organize  the  data  using  the  sample  worksheet  in 
Appendix  B,  Figure  B.l. 
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3.  Even  if  a licensing  contract  or  consortium  arrangement  period  is  different 
from  the  reporting  period,  use  the  annual  licensing  fee  to  calculate  the 
statistics. 


4.  If  a fee  is  paid  to  a consortium  or  other  joint  arrangement,  include  the 
amount.  In  the  case  where  a fee  is  paid  for  an  aggregate  service  and  the 
service  contains  different  categories  of  resources  (full-text  journals  and 
reference  sources)  as  a bundle,  use  an  estimate  based  on  expected  or 
historical  use,  or  list  prices. 

5.  Note  any  major  commitments  (such  as  JSTOR  one-time  costs)  that  do  not 
occur  year  to  year  and  that  significantly  influence  the  reported  amount. 

Special  considerations:  Whereas  the  patron  accessible  resource 
counts  reflect  the  extensiveness  of  electronic  resources  at  a given  point  in  time 
(most  likely  at  the  end  of  the  reporting  period,  be  it  a month  or  a year),  the  cost 
figures  cover  a period  of  time.  Ideally  the  amount  of  money  spent  reflects  the 
number  and  extensiveness  of  resources.  Prorating  licensing  fees  addresses  part  of 
the  problem  of  matching  the  resources  with  the  money  spent,  but  can  be  very  time 
consuming.  You  can  report  the  annual  amount  paid  without  prorating  on  the  basis 
that  over  the  years  the  figures  will  even  out. 


Some  electronic  full-text  journals  come  either  as  a free  service  with  a print 
subscription  or  as  part  of  a print-plus-online-access  subscription  (the  library  pays 
extra  for  electronic  access).  In  the  first  case,  the  problem  is  whether  or  not  to  post 
any  amount  for  the  cost  of  electronic  access.  In  the  latter  case,  the  question  is  how 
much  of  the  cost  can  be  attributed  to  electronic  access. 


Cost  of  electronic  reference  sources 


Definition:  Expenditures  for  electronic  reference  sources  and  aggre- 

gate services  that  the  library  provides  to  users  either  through  individual  licensing 
contracts  with,  content  providers  or  through  consortia  or  other  arrangements  where 
the  library  pays  some  fees.  These  fees  include  both  annual  access  fees  and  other 
service  costs  paid  to  the  vendor  directly  or  through  consortial  arrangements. 

Rationale:  This  statistic,  cost  of  electronic  reference  sources,  was 
developed  by  ARL  to  determine  how  much  libraries  are  spending  on  electronic 
reference  sources  and  how  new  forms  of  electronic  reference  sources  are  replacing 
traditional  reference  materials.  It  also  gives  insight  into  shifts  in  budget  alloca- 
tions from  print  to  electronic  materials,  or  new  allocations  exclusively  for  electronic 
materials.  Furthermore,  this  statistic  allows  libraries  to  calculate  unit  costs  of 
electronic  reference  sources  after  collecting  C2  and  R2  figures.  This  figure  assists 
libraries  in  making  decisions  about  how  effectively  they  are  serving  their  potential 
and  intended  audience,  and  in  benchmarking  with  other  institutions. 

Implementation 

□ Collected  by:  Local 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent 
(e.g.,  monthly,  quarterly) 
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□ Procedures:  For  the  definition  of  electronic  reference  sources,  please  refer  to  the 
definition  of  R2.  For  libraries  that  do  not  have  acquisitions  systems  which 
support  coding  of  materials  expenditures  by  the  categories  used  in  the  manual,  it 
may  be  necessary  to  create  an  in-house  spreadsheet  or  database  file  to  keep  track 
of  cost  information  according  to  the  types  of  resources  (e-journals,  reference 
databases,  and  e-books).  Preferably  a single  file  will  contain  contract  informa- 
tion (duration,  cost),  updated  title  counts,  and  reported  usage  statistics. 

1.  Gather  reports  and  invoices  related  to  electronic  databases  and  resources 
for  the  reporting  period.  These  documents  are  typically  handled  by  the 
library’s  accounting  office. 

2.  If  you  have  not  done  so,  organize  the  data  using  the  sample  worksheet  in 
Appendix  B,  Figure  B.l. 

3.  Even  if  a licensing  contract  or  consortium  arrangement  period  is  different 
from  the  reporting  period,  use  the  annual  licensing  fee  to  calculate  the 
statistics. 

4.  If  a fee  is  paid  to  a consortium  or  through  other  joint  arrangement,  include 
the  amount.  If  a fee  is  paid  for  an  aggregate  service  and  the  service 
contains  different  categories  of  resources  (full-text  journals  and  reference 
sources)  as  a bundle,  use  an  estimate  based  on  expected  or  historical  use,  or 
list  prices. 

5.  In  the  comments  field  of  the  sample  worksheet  (Appendix  B,  Figure  B.l), 
report  any  major  commitments  that  do  not  occur  year  to  year  and  that 
significantly  influence  the  reported  amount. 

Sp  ecial  considerations:  Whereas  the  patron  accessible  resource 
counts  reflect  the  extensiveness  of  electronic  resources  at  a given  point  in  time 
(most  likely  at  the  end  of  reporting  period,  be  it  a month  or  a year),  the  cost  figures 
cover  a period  of  time . Ideally  the  amount  of  money  spent  reflects  the  number  and 
extensiveness  of  resources.  Prorating  licensing  fees  addresses  part  of  the  problem 
of  matching  the  resources  with  the  money  spent,  but  can  be  very  time  consuming. 
You  can  report  the  annual  amount  paid  without  prorating  on  the  basis  that  over  the 
years  the  figures  will  even  out. 


Cost  of  electronic  books 


Definition:  Expenditures  for  electronic  full-text  monographs  that  the 
library  offers  to  its  users.  Include  both  initial  purchase  costs  and  membership  fees 
as  well  as  annual  access  and  service  fees  paid  directly  or  through  consortia  arrange- 
ments. 

Rationale:  This  statistic,  cost  of  electronic  books,  was  developed  by 
ARL  to  determine  how  much  libraries  were  spending  on  electronic  books.  It  also 
gives  an  idea  about  the  extent  of  budget  allocations  for  electronic  resources.  Fur- 
thermore, this  statistic  allows  libraries  to  calculate  unit  costs  of  e-books  after 
collecting  C3  and  R3  statistics,  aids  them  in  determining  how  effectively  they  are 
serving  their  potential  and  intended  audiences,  and  assists  them  in  benchmarking 
with  other  institutions. 
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Implementation 


□ Collected  by:  Local 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent 
(monthly,  quarterly) 

□ Procedures:  For  the  definition  of  electronic  books,  please  refer  to  the  definition 
of  R3.  Current  library  accounting  systems  generally  do  not  support  coding  of 
materials  expenditure  by  the  categories  used  in  the  manual.  Therefore,  it  may 
become  necessary  to  create  an  in-house  spreadsheet  or  database  file  to  keep 
track  of  cost  information  according  to  the  types  of  resources  (e-journals,  refer- 
ence databases,  and  e-books).  Preferably  a single  file  will  contain  contract 
information  (duration,  cost),  updated  title  counts,  and  reported  usage  statistics. 

1.  Gather  reports  and  invoices  related  to  electronic  databases  and  resources 
for  the  reporting  period.  These  documents  are  typically  handled  by  the 
library’s  accounting  office.  You  may  also  need  to  review  circulation  records 
to  verify  the  accuracy  of  invoices  if  additional  per-use  fees  are  paid  (royalty 
on  use,  as  with  E-reserves). 

2.  If  you  have  not  done  so,  organize  the  data  using  the  sample  worksheet  in 
Appendix  B,  Figure  B.l. 

3.  Even  if  a licensing  contract  or  consortium  arrangement  period  is  different 
from  the  reporting  period,  use  the  annual  licensing  fee  to  calculate  the 
statistics. 


4.  If  a fee  is  paid  to  a consortium  or  other  joint  arrangement,  include  the 
amount. 

5.  Note  any  major  commitments  (such  as  netLibrary  purchase  costs)  that  do 
not  occur  year  to  year  and  that  significantly  influence  the  reported  amount. 

Special  considerations:  Whereas  the  patron  accessible  resource 

counts  reflect  the  extensiveness  of  electronic  resources  at  a given  point  in  time 
(most  likely  at  the  end  of  the  reporting  period,  be  it  a month  or  a year),  the  cost 
figures  cover  a period  of  time.  Ideally  the  amount  of  money  spent  reflects  the 
number  and  extensiveness  of  resources.  Prorating  licensing  fees  addresses  part  of 
the  problem  of  matching  the  resources  with  the  money  spent,  but  it  can  be  very 
time  consuming.  You  can  report  the  annual  amount  paid  without  prorating  with 
the  rationale  that  over  the  years  the  figures  will  even  out. 


Traditionally  books  are  purchased  on  a one-time  payment  in  exchange  for 
permanent  ownership  by  the  library.  However,  with  regard  to  electronic  books,  it 
appears  that  some  arrangements  allow  libraries  to  subscribe  to  an  e-book  collection 
at  a predetermined  fee  and  for  a predetermined  interval  of  time.  We  are  concerned 
with  the  format  of  the  material,  not  the  subscription  or  payment  arrangement. 
These  materials  should  be  counted  as  books,  not  serial  publications. 


Related  issues:  In  many  instances,  the  physical  form  of  the  material 
(print,  electronic)  may  change  the  nature  of  the  object.  An  electronic  book  is  a 
good  example.  With  enhancements  such  as  full-text  searching  (although  print 
books  too  have  some  search  capability  through  tables  of  contents  and  indexes), 
electronic  books  support  new  forms  of  searching  not  present  in  print. 
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Library  expenditures  for  bibliographic 

UTILITIES,  NETWORKS  AND  CONSORTIA 


Definition:  Expenditures  paid  by  the  library  for  services  provided  by 
national,  regional,  and  local  bibliographic  utilities,  networks,  and  consortia  such  as 
OCLC,  RLG,  excluding  fees  paid  for  user  database  access  and  subscriptions}  which 
should  be  reported  in  Cl  through  C3. 

Rationale:  This  statistic  is  based  on  the  ARL  Supplementary  Statistics 
Survey.  It  was  developed  by  ARL  to  determine  how  much  money  libraries  spend 
for  bibliographic  utilities,  networks,  and  consortia.  Because  individual  libraries 
often  have  to  deal  with  special  provisions  and  funding  issues  related  to  contracts, 
this  statistic  may  not  lend  itself  to  comparability  among  ARL  member  libraries. 
Nevertheless,  it  represents  an  attempt  to  keep  track  of  the  financial  relationships 
between  bibliographic  utilities  and  libraries.  Although  this  may  provide  very 
limited  comparability,  it  is  an  estimate  of  the  cost  of  bibliographic  utilities,  net- 
works, and  consortia. 

Implementation 

□ Collected  by:  Local 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more  fre- 
quent (e.g.,  monthly,  quarterly) 

□ Procedures: 

1.  Gather  reports  and  invoices  with  bibliographic  utilities,  networks,  and 
consortia  of  which  the  library  is  a member  for  the  whole  or  part  of  the 
reporting  period.  These  documents  are  typically  handled  by  the  library's 
accounting  office. 

2.  Identify  only  those  expenditures  paid  to  the  bibliographic  utilities,  net- 
works, and  consortia  for  membership,  maintenance,  and  other  infrastruc- 
ture. Do  not  include  expenditures  that  are  directly  attributable  to  access  of 
electronic  resources.  Those  expenditures  should  be  included  in  Cl  through 
C3. 

For  instance,  if  your  library  paid  a total  of  $100,000  to  OCLC  for  its 
various  services  and  your  best  guess  of  electronic  database  access  portion  of 
the  services  is  80%,  then  you  should  report  $80,000  for  C2  and  the  re- 
maining $20,000  for  C4. 

3.  Even  if  a membership  or  consortium  period  is  different  from  the  reporting 
period,  use  the  amount  of  the  membership  or  consortium  agreement. 

4.  Use  the  sample  form  in  Appendix  B,  Figure  B.2  to  compile  the  expendi- 
tures. 

Special  consideration  s:  Prorating  can  be  time  consuming.  Consor- 
tia or  other  memberships  may  bring  additional  benefits,  such  as  subscriptions, 
training  or  preferential  pricing  for  acquisition  of  materials.  It  may  be  difficult  to 
separate  pure  membership  fees  from  value-added  services  of  membership  (e.g., 
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original  catalog  credits  from  OCLC  that  may  be  used  to  offset  costs  of  databases, 
purchase  of  catalog  records,  etc.).  Report  the  annual  amount  paid  without  prorat- 
ing with  the  rationale  that  over  the  years  the  figures  will  even  out. 


External  expenditures  for  bibliographic 

UTILITIES,  NETWORKS,  AND  CONSORTIA 
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Definition:  Expenditures  paid  by  external  agencies,  such  as  state 
government  agencies,  on  the  library's  behalf  for  access  to  computer  files,  electronic 
serials,  or  search  services  through  a centrally  funded  system  or  consortial  arrange- 
ments. Examples  include  state-  (or  province-)  supported  networks  such  as  VIVA 
(Virginia),  CNSLP  (Canadian  National  Site  Licensing  Project),  and  the  University 
of  California’s  California  Digital  Library  Expenditure. 

Rationale:  Like  statistic  C4,  this  statistic  is  based  on  the  ARL  Supple- 
mentary Statistics  Survey.  It  was  developed  by  ARL  to  determine  how  much  money 
is  spent  for  bibliographic  utilities,  networks,  and  consortia  on  libraries’  behalf  for 
access  to  computer  files,  serials,  and/or  services  through  consortial  arrangements. 
Because  of  contractual  issues,  this  statistic  may  provide  little  comparability  among 
ARL  member  libraries.  Nevertheless,  it  can  give  ARL  members  an  estimate  of  the 
external  costs  of  bibliographic  utilities,  networks,  and  consortia. 


Implementation 

□ Collected  by:  Local  and  external  bodies  such  as  regional  and  academic  consortia 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent 
(e.g.,  monthly,  quarterly). 

□ Procedures: 


1.  Gather  reports  and  invoices  with  bibliographic  utilities,  networks,  and 
consortia  that  are  related  to  electronic  databases  and  resources  for  the 
reporting  period.  These  documents  are  typically  handled  by  the  library’s 
accounting  office.  However,  they  can  be  maintained  outside  the  organiza- 
tion and,  in  some  instances,  may  only  be  provided  to  libraries  upon  de- 
mand. 

2.  Find  out  how  much  of  the  central  funding  is  attributable  to  your  library. 

For  example,  if  your  library  contributes  a total  of  $60,000  over  a period  of 
three  years  to  a state  consortium  that  has  a matching  contribution  of 
$120,000  for  the  same  period,  the  amount  to  report  as  C5  for  a given  year 
during  the  three-year  period  will  be  $40,000  ($120,000  x 1/3).  The 
library’s  contribution  ($60,000)  has  to  be  divided  annually  and  posted  in 
Cl  through  C3. 

If  the  specific  dollar  amount  is  not  known,  but  the  total  student  FTE  (full- 
time equivalent)  for  the  consortium  and  the  amount  spent  for  the  academic 
members  are  known,  divide  the  overall  amount  spent  by  your  institution’s 
share  of  the  total  student  FTE.  Alternatively,  if  the  consortium  is  com- 
prised of  different  types  of  institutions  (academic,  public,  or  corporate), 
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but  the  library  has  information  about  the  portion  of  its  own  use  among  the 
consortium  participants,  multiply  the  total  amount  by  the  percentage  of 
known  (or  estimated)  usage  rate. 

3.  As  a last  resort,  consult  with  a staff  member  overseeing  the  consortium  or 
the  central  funding  system  to  get  an  estimate  of  the  portion  of  the  central 
funding  that  is  attributable  to  the  library.  Please  make  a note  of  this  in  the 
comments  field  in  the  sample  worksheet  (Appendix  B,  Figure  B.3). 

4.  Use  the  sample  form  in  Appendix  B Figure  B.3  to  compile  the  expendi- 
tures. 
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Statistics  Related  lo  Library  Digilizalion  Activities 


Comprised  of  resource  and  use  measures,  the  digital  collection  measures 
attempt  to  describe  where  libraries  are  in  creating  and  making  available 
local  (perhaps  unique)  content  that  may  not  have  been  previously  accessible.  Such 
collections  can  attract  students  and  faculty  to  your  university  and  thereby  enhance 
the  institution’s  reputation.  As  more  libraries  digitize  resources,  more  users  will  be 
able  to  retrieve  those  unique  resources  at  anytime  and  from  anywhere.  Digital 
library  projects,  as  well  as  other  network  resources  and  services,  also  will  serve 
increasing  numbers  of  students  taking  courses  online. 

Collecting  library  digitization  measures  may  provide  an  opportunity  for 
benchmarking  and  may  encourage  libraries  to  devote  more  time  and  allocate  more 
resources  to  this  worthwhile  endeavor.  It  should  be  noted  that  these  statistics 
represent  a very  early  attempt  to  measure  digitization  of  resources;  as  time  passes 
and  the  technology  advances,  some  of  the  definitions  and  procedures  may  need  to 
be  revisited  and  modified.  During  the  field-testing  it  was  reported  that  storing 
and  maintaining  digitized  resources  had  been  an  issue.  The  unavailability  of  an 
appropriate  infrastructure  in  some  institutions  meant  that  the  project  did  not 
include  statistics  related  to  library  digitization  projects. 

Libraries  archive  the  scholarly  output  of  their  institutions  - theses  and 
dissertations  - in  both  paper  and  digital  form.  Digital  collections  also  provide  new 
opportunities  with  faculty  to  archive  research  results.  These  statistics,  although 
preliminary,  form  a basis  for  tracking  these  issues. 


□ D1  Size  of  library  digital  collection 

□ D2  Use  of  library  digital  collection 

□ D3  Cost  of  digital  collection  construction  and 
management 


Library 

Digitization 

Activities 


Size  of  Library  digital  collection 


Definition:  Library  digital  collection  refers  to  digital  materials  (texts, 
images,  and  audio-visuals)  created  in  or  converted  from  different  formats  (e.g., 
paper,  microfilm,  tapes,  etc.)  by  the  library  and  made  available  to  users  electroni- 
cally. This  includes  electronic  theses  and  dissertations  (ETDs),  special  collections 
materials,  maps,  sound  recordings,  films,  and  other  digital  materials  that  are  not 
purchased  or  acquired  from  outside  through  individual  or  consortial  licensing 
agreements.  It  includes  the  number  of  titles  and  size  (in  gigabytes)  by  sub-catego- 
ries (ETD,  visual  materials,  texts,  multimedia),  and  as  an  aggregate  at  the  end  of 
the  reporting  period.  It  also  includes  the  number  of  items  (titles)  added  during  the 
reporting  period. 
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The  types  of  formats  in  Appendix  B,  Figure  B.4,  refer  to  original  formats 
rather  than  the  digitized  outputs.  Examples  of  visual  materials  include  photos, 
maps,  and  postcards.  Examples  of  text  include  books,  journal  articles  and  pam- 
phlets. Examples  of  multimedia  include  audio,  video,  and  other  interactive  materi- 
als. However,  this  statistic  does  not  include  any  back  up  copies  or  mirror  sites 
because  items  should  be  counted  only  once. 

Rationale:  Collecting  library  digitization  measures  may  provide  an 
opportunity  for  benchmarking  in  terms  of  file  sizes  for  the  resources  that  have 
been  digitized.  Moreover,  the  statistic  can  demonstrate  that  libraries  are  not 
merely  brokers  of  external  information  resources,  but  also  producers  of  information 
content  and  useful  finding  aids. 

This  statistic  provides  information  on  the  extent  of  digital  library  projects, 
the  life  cycle  of  such  projects,  and  the  “virtual  space”  requirements  of  such  collec- 
tions. 


Implementation 

□ Collected  by:  Local 

O Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent 
(e.g.,  monthly,  quarterly) 

□ Procedures:  For  cases  in  which  multiple  digital  formats  (derivatives)  were 
produced  from  an  item,  count  it  only  once  based  on  the  type  of  item  that  was 
digitized.  For  example,  if  a 100-page  book  was  digitized  in  100  TIFF  files,  each 
containing  a page,  a 100-page  PDF  file,  and  10  PDF  files  (one  PDF  file  for  each 
of  10  chapters),  count  it  as  a single  text  with  100  pages.  If  a derivative  item 
was  used  as  the  source,  do  not  count  the  outputs.  But  in  the  total  size  (in 
gigabytes)  include  all  versions  of  derivatives. 

1.  Designate  a staff  member  to  coordinate  the  collection  of  this  statistic.  The 
person  should  be  well  aware  of  library  digital  collection  activities. 

2.  Identify  library  staff  in  charge  of  various  digital  library  projects  and 
initiatives. 

3.  It  is  necessary  to  conduct  an  inventory  of  digital  material  stock  using  the 
sample  tally  worksheet  in  Appendix  B,  Figure  B.6  if  it  has  not  been  done 
already.  If  this  inventory  information  is  already  available,  enter  it  into  the 
worksheet.  When  the  inventory  is  completed,  summarize  the  information 
using  the  sample  worksheet  in  Appendix  B,  Figure  B.4.  Add  additional 
categories  if  necessary. 

4.  After  obtaining  the  inventory  information,  ask  staff  members  to  keep  track 
of  additional  output  regularly  using  the  sample  tally  worksheet  in  Appen- 
dix B,  Figure  B.6. 

5.  At  the  end  of  the  reporting  period,  collect  the  worksheets  and  calculate  the 
total  production  during  the  reporting  period  using  the  worksheet  in 
Appendix  B,  Figure  B.5.  Add  additional  categories  if  necessary. 


28 


4 Gy 


Related  issues:  Realistically,  each  digital  collection  is  unique  in  terms 
of  the  production  process,  the  way  it  is  intended  to  be  used,  its  focus,  and  mainte- 
nance. It  is  important  to  use  appropriate  units  of  measurement  to  describe  the 
overall  size  and  extensiveness  of  the  whole  collection. 


Because  of  the  wide  variations  of  the  types  and  features  of  digital  collections 
constructed  at  ARL  institutions,  this  statistic  may  be  more  useful  locally  than  for 
comparison  across  ARL  member  libraries.  Benchmarking  may,  however,  be  possible 
from  the  data  collected  to  produce  some  qualitative  and  quantitative  indicators  as 
to  the  extent  of  digital  library  collection  activities  and  different  emphases  across  “X 

the  ARL  membership.  ^ 


Use  of  library  digital  collection 


Definition:  Number  of  times  library  digital  collection  titles  and 
physical  files  were  accessed  and  the  number  of  searches  (queries)  conducted  (if 
there  is  such  a capability)  during  the  reporting  period. 

Rationale:  Each  digital  collection  is  unique  in  terms  of  its  focus, 
production  process,  and  the  ways  it  is  intended  to  be  used  and  maintained.  There- 
fore, because  of  the  wide  variations  of  the  types  and  features  of  these  library 
collections  constructed  at  different  ARL  institutions,  this  statistic  needs  to  be 
collected  and  used  locally  instead  of  across  ARL  member  libraries.  Nevertheless, 
this  statistic  has  the  potential  to  produce  some  qualitative  and  quantitative  indica- 
tors as  to  how  these  collections  are  being  used  and  serving  the  intended  user 
community’s  needs. 

Implementation 

□ Collected  by:  Local 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more  frequent 
(e.g.,  monthly,  quarterly) 

□ Procedures: 

1.  Designate  a staff  member  to  coordinate  the  collection  of  this  statistic.  The 
person  should  be  well  versed  in  the  use  of  web  log  software  and/or  statis- 
tics provided  by  the  software.  This  person  will  act  as  a liaison  for  staff 
members  who  are  responsible  for  managing  library  digital  collections. 
Obtaining  the  statistic  may  require  some  level  of  programming  (e.g.,  Unix 
scripting  and  SQL). 

2.  Items  accessed  can  be  collected  in  various  ways,  and  depending  on  your 
library’s  environment,  your  library  may  need  to  collect  different  access 
statistics. 

Although  you  are  asked  to  collect  both  title  access  and  physical  file  access, 
if  it  takes  too  much  time  and  effort  to  collect  the  title  access,  report  the 
physical  file  access  count  only.  For  example,  a book  can  be  digitized  and 
made  into  10  PDF  files,  each  containing  a chapter,  for  access.  Suppose  a 
user  viewed  five  PDF  files  out  of  10.  In  this  case,  you  will  have  five 
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physical  item  accesses  and  one  title  access.  Usually  it  is  easier  to  have  a 
physical  item  access  count,  while  it  takes  custom  programming  to  compute 
the  title  access  count  as  most  off-the-shelf  web  traffic  software  packages 
do  not  provide  this. 

Do  not  report  web  page  hits.  Instead,  count  how  many  times  the  digitized 
items  were  accessed  (the  exact  name  for  item  access  may  vary  depending  on 
the  type  of  web  traffic  analysis  software  being  used  in  the  library). 

If  a search  capability  is  a feature  of  a library  digital  collection,  the  total 
number  of  searches  submitted  needs  to  be  collected.  A search  represents 
an  explicit  user  request  for  specific  information  in  a database  and  is  ex- 
pressed usually  in  the  form  of  word  strings.  Clicks  on  web  page  buttons, 
such  as  “Next”  and  “Previous,”  do  not  count  as  user  searches. 

You  might  want  to  install  web  traffic  analysis  software  (e.g.,  WebTrends, 
Web  Tracks)  on  the  library  web  servers  housing  library  digital  collection 
materials,  if  the  web  servers  do  not  have  such  software  already.  You  might 
want  to  consider  installing  a trial  version  that  gives  between  30-180  days 

- i of  free  trial. 

- Read  the  description  of  reported  statistics  carefully  and  make  sure  that  the 
software  provides  what  you  want. 

3.  If  continuous  collection  of  use  statistics  is  not  possible  or  desirable,  select  a 
typical  week  (or  month)  to  run  a sample  study.  Be  sure  to  vary  the  specific 
. ,.t  week  (or  month)  chosen  over  the  course  of  a year  or  from  year  to  year  to 

• y account  for  seasonal  fluctuations.  Extrapolate  based  on  the  sample  data. 

I 4.  At  the  end  of  the  report  period,  use  the  log  analysis  report  to  calculate  the 

number  of  accesses  to  library  digital  collection  items.  Use  the  sample 
report  in  Appendix  B,  Figure  B.7,  to  organize  the  information. 

Special  considerations:  To  the  extent  possible,  exclude  accesses  by 
web  search  spiders.  Also,  do  not  include  accesses  to  auxiliary  (or  incidental)  items 
that  are  not  part  of  the  library  digital  collection  content  (.gif  buttons  and  image 
- )v  maps  for  navigation).  Note  the  method  used  and  include  a description  of  any 
filtering  done. 

Related  issues:  This  statistic  needs  to  be  collected  and  used  locally 

/ instead  of  across  ARL  member  libraries  because  of  the  wide  variations  of  the  types 
and  features  of  digital  collections  constructed  at  ARL  institutions. 


Cost  of  digital  collection 

CONSTRUCTION  AND  MANAGEMENT 


Definition:  Annual  direct  costs  (personnel,  equipment,  software, 
contracted  services  and  similar  items)  spent  to  create  digital  materials  (texts, 
images,  and  multimedia)  or  to  convert  existing  materials  into  digital  form  for  the 
purpose  of  making  them  electronically  available  to  users.  Include  expenditures 
related  to  digitization,  OCR,  editorial,  creation  of  markup  texts,  preparation  of 
metadata  for  access  to  digitized  materials,  data  storage,  and  copyright  clearance. 
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Exclude  expenditures  for  information  resources  purchased  or  acquired  from  outside 
the  institution  through  individual  or  consortial  licensing  agreements. 

Rationale:  The  cost  of  each  digital  collection  construction  may  vary 
significantly,  depending  on  the  size  of  the  collection,  conditions  of  the  sources 
before  digitizing,  available  infrastructure,  staff  allocation,  timeline,  and  administra- 
tive support.  This  statistic  should  be  collected  and  used  locally  instead  of  across 
ARL  member  libraries  because  of  the  wide  variability  among  these  library  collec- 
tions constructed  at  different  ARL  institutions.  Nevertheless,  this  statistic  has  the 
potential  to  provide  quantitative  indicators  as  to  how  costly  these  efforts  are,  how 
much  resource  allocation  (i.e.,  budget  allocation,  staffing,  infrastructure,  etc.)  is 
needed,  and  how  well  they  serve  the  intended  user  community’s  needs  (e.g.,  to 
account  for  internal  and  external  costs  to  construct  and  manage  digital  collections 
at  ARL  libraries). 

Implementation 

□ Collected  by:  Local 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more  fre- 
quent (e.g.,  monthly,  quarterly) 

□ Procedures: 

1.  Designate  a staff  member  to  coordinate  the  collection  of  this  statistic. 

2.  Direct  the  designated  staff  member  to  contact  library  staff  members  who 
are  in  charge  of  digital  collection  projects.  Ask  all  library  staff  members 
involved  in  any  digital  collection  projects  as  part  of  their  official  responsi- 
bilities to  fill  out  the  worksheet  in  Appendix  B,  Figure  B.8,  for  the  report- 
ing period.  Ask  them  to  estimate  how  much  of  their  time  was  spent  on 
planning,  implementing,  and  managing  digital  collection  projects.  This 
information  will  be  entered  in  the  worksheet  as  FTE.  A further  break- 
down of  activities  may  be  necessary  if  the  library  wants  to  have  more 
detailed  information  on  the  distribution  of  efforts. 

Note  that  annual  salary  should  not  be  asked  of  the  staff  members  filling 
out  the  worksheet  and  should  not  include  fringe  benefits.  When  all  the 
worksheets  are  collected,  the  salary  information  will  be  obtained  from  the 
library  accounting  or  personnel  department.  Direct  staff  cost  will  then  be 
calculated. 

The  personnel  cost  should  also  include  wages  paid  to  non-salaried  staff, 
including  student  and  other  hourly  workers. 

3.  Cost  of  equipment  should  be  amortized.  For  example,  if  a S3, 000  scanner 
was  purchased  at  the  beginning  of  the  reporting  year  and  has  a deprecia- 
tion period  of  three  years,  register  SI, 000  as  the  equipment  cost.  Costs  of 
software  should  be  reflected  in  full  amounts  based  on  the  time  of  the 
purchase. 

4.  If  a subcontracting  period  is  different  from  the  reporting  period,  prorate 
the  amount  for  the  reporting  period.  If  the  payment  is  based  on  percent  to 
completion,  include  only  the  amount  that  belongs  to  the  reporting  period. 

5.  Use  the  sample  worksheet  in  Appendix  B,  Figure  B.9,  to  calculate  the  total 
cost. 
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Related  issues:  This  statistic  needs  to  be  collected  and  used 

locally  instead  of  across  ARL  member  libraries  because  of  the  wide  variations  of 
the  types  and  features  of  digital  collections  constructed  at  ARL  institutions. 


o 

ERLC 


32 


411 


Analysis  of  Suggested  Performance  Measures 


The  overall  rationale  for  the  performance  measures  in  this  study  is  to 
provide  a means  for  measuring  the  proportion  of  services  delivered 
through  traditional  channels  relative  to  analogous  services  delivered  through 
electronic  channels.  These  measures  will  help  document  trends  in  service  delivery 
for  the  purpose  of  allocating  staff  and  development  resources  as  well  as  identify 
trends  for  strategic  planning  of  service  delivery. 


□ Pi  Percentage  of  electronic  reference  transactions  of  total 
reference 

O P2  Percentage  of  virtual  library  visits  of  all  library  visits 
O P3  Percentage  of  electronic  books  to  all  monographs 


Performance 

Measures 


O Percentage  of  Electronic  Reference 
Transactions  of  Total  Reference 


Definition:  Percentage  of  annual  electronic  reference  transactions  to 
total  reference  transactions.  An  electronic  reference  transaction  must  include  a 
question  either  received  electronically  (e.g.,  via  e-mail,  WWW  form,  etc.)  or 
responded  to  electronically.  Count  excludes  phone  and  fax  traffic  unless  either  the 
question  or  answer  transaction  occurs  via  the  described  manner.  It  includes  the 
counts  accrued  from  participation  in  any  local  and  national  projects,  such  as 
DigiRef  and  the  Library  of  Congress’s  CDRS  (Collaborative  Digital  Reference 
Service). 


Total  reference  = Traditional  reference  counts  (include  face-to-face  reference 
transactions,  telephone  and  fax  reference  counts)  + electronic  reference  transaction 
counts. 


U1(p.11) 


Total  reference  transactions 


• 100 


Rationale:  The  purpose  of  having  a networked  environment  is  to 
promote  connectedness.  This  measure  provides  an  indication  of  a changing  library 
environment.  While  in  the  traditional  library  environment  reference  transactions 
were  handled  mainly  through  non-electronic  means,  in  the  current  environment 
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reference  transactions  can  be  handled  via  various  electronic  means  over  the 
Internet.  By  having  this  as  a measure,  libraries  are  able  to  track  the  development 
of  a new  and  emerging  library  service  and  have  a number  that  fully  represents 
reference  activities.  This  measure  may  indicate  how  often  various  electronic 
applications  are  used  in  any  given  period  and  also  assist  decision-makers  in  reallo- 
cating resources.  Moreover,  this  performance  measure  will  give  administrators 
trend  data  on  how  network  services  are  being  used  and  this  data  can  then  be  used 
for  future  planning. 

Implementation 


□ Collected  by:  Local 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more 

frequent  (monthly,  quarterly) 

□ Procedures: 


1.  If  continuous  collection  of  this  statistic  is  not  possible  or  desirable,  select  a 
typical  week  (or  month)  to  run  a sample  study.  It  is  recommended  that  you 
sample  a week  in  a different  month  or  several  months  to  account  for 
seasonal  fluctuations.  Extrapolate  based  on  the  sample  data. 

2.  Designate  a staff  member  to  coordinate  the  collection  of  this  measure.  Key 
tasks  include  distributing  a daily  tally  sheet,  collecting  the  daily  tally  sheet, 
adding  each  day’s  total  to  a weekly  Figure,  and  being  available  to  respond  to 
data  collection  problems  should  they  arise. 


3.  For  electronic  transactions,  use  the  count  obtained  by  following  the  proce- 
dures for  U1  (p.  11). 


4.  Total  the  overall  number  of  transactions. 

5.  Divide  the  number  of  electronic  reference  transactions  by  the  total  number 
of  transactions. 


6.  Multiply  by  100. 

7.  Indicate  and  describe  any  additional  methods  used  outside  of  this  defini- 
tion and  these  guidelines. 


Special  considerations:  Count  the  number  of  transactions,  not  the 

number  of  questions.  That  is,  if  one  request  is  emailed  with  three  questions,  it 
should  be  counted  as  one  transaction,  not  three. 


Percentage  of  Virtual  Library  Visits 
of  All  Library  Visits 


Definition:  Number  of  virtual  library  visits  out  of  all  library  visits. 

A virtual  library  visit  is  when  a user  visits  the  library’s  website  or  catalog  for 
any  length  of  time  or  for  any  purpose  from  outside  the  physical  plant  of  the  library, 
regardless  of  the  number  of  pages  or  items  viewed  or  requested.  The  term  “virtual 
visit”  excludes  in-library  visits  where  a patron  or  a staff  member  uses  electronic 
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resources.  If  a user  looked  at  16  pages  and  54  graphic  images  while  at  a website, 
that  user  registers  one  visit  on  the  web  server.  A visit  is  usually  determined  by  a 
user’s  IP  address.  Due  to  various  server  management  issues  and  differing  software, 
this  measure  is  an  estimate  of  the  visits  to  the  library  site. 

All  library  visits  is  the  total  of  the  number  of  virtual  library  visits  plus  the 
number  of  physical  visits  to  the  library  including  branches. 


U5  (p.  16) 


100 


Total  Library  Visits 


Rationale:  People  accessing  the  website  or  catalog  from  outside  the 
library  will  reflect  interest  in  library  services.  The  idea  of  having  network  services 
is  to  expand  the  reach  of  libraries  beyond  their  physical  boundaries,  and  this 
performance  measure  can  provide  information  about  how  far  network  services  are 
reaching.  This  figure  will  also  show  the  use  of  the  library  outside  the  regular  place 
of  business,  which  will  be  a more  accurate  depiction  of  library  use.  Having  this 
measure  is  important  to  show  the  continued  relevance  of  library  service  if  physical 
attendance  figures  decrease. 

Implementation 

□ Collected  by:  Local  and/or  vendors 

D Frequency:  Reported  annually,  although  internal  reporting  may  be  more 

frequent  (monthly,  quarterly) 

□ Procedures: 

1.  Obtain  the  virtual  library  visits  count  using  the  procedures  for  U5  (virtual 
library  visits). 

2.  Obtain  physical  attendance  count  from  turnstile  counts  or  swipe  card 
records.  To  the  extent  possible,  collect  comprehensive  data  from  all  library 
branches. 

3.  Combine  the  virtual  visit  count  and  the  physical  attendance  count. 

4.  Divide  the  number  of  virtual  library  visits  by  the  total  library  visits. 

5.  Multiply  by  100.  For  example,  a library  had  1,000  external  virtual  visits 
and  9,000  physical  visits  for  a total  visit  composite  measure  of  10,000. 
1,000  virtual  visits  divided  by  10,000  total  visits  equals  .10  (or  10%). 


Percentage  of  Electronic  Books  to 
All  Monographs 


:;C; 


Definition:  Percentage  of  the  number  of  electronic  books  available  to 
users  (through  either  an  individual  licensing  contract  or  other  consortial  arrange- 
ments) to  all  the  library’s  monographs. 
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Rationale:  Networking  technology  in  libraries  continues  to  improve, 
thereby  increasing  user  access  to  electronic  counterparts  of  some  traditional 
sources.  In  the  mid  90s,  networking  and  resource  sharing  technologies  facilitated 
print  and  e-book  access  through  library  networks.  This  performance  measure 
attempts  to  document  the  degree  of  expansion  of  e-books  to  all  monographs. 
Libraries  should  use  caution  while  collecting  this  measure  because  the  definition  of 
e-books  is  itself  still  evolving.  This  statistic  is  an  early  attempt  to  keep  track  of 
this  type  of  source  that  is  becoming  increasingly  available. 

Implementation 

□ Collected  by:  Local  and  vendors 

□ Frequency:  Reported  annually,  although  internal  reporting  may  be  more  fre- 

quent (monthly,  quarterly) 

□ Procedures: 

1.  Identify  all  types  of  monographic  materials.  Use  in-house  record-keeping 
sources  and  other  library  sources  to  determine  the  number  of  all  mono- 
graphs, including  electronic  books,  non-electronic  books,  and  other  mono- 
graphic materials. 

2.  Identify  electronic  book  types,  including  electronic  books  and  electronic 
full-text  aggregate  services,  using  the  sources  in  step  1 of  the  procedures 
for  R3. 

3.  Count  individual  electronic  book  titles.  Record  the  number  of  individual 
electronic  books  from  the  spreadsheet  or  record  the  number  from  another 
source. 

4.  Exclude  electronic  reference  books,  i.e.,  publicly  available  electronic  books 
that  are  accessed  for  free. 

5.  Calculate  the  total  number  of  all  monographs,  including  electronic  books, 
non-electronic  books,  and  other  monographic  materials. 

6.  Divide  the  number  of  electronic  books  by  the  number  of  all  library  mono- 
graphs (electronic  and  non-electronic  monographs). 

7.  Multiply  by  100. 

8.  Indicate  and  describe  any  additional  methods  used  outside  of  this  defini- 
tion and  guidelines. 
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Improving  Networked  Statistics 

With  the  ever-increasing  portion  of  library  collections'  dollars  committed  to 
networked  services,  there  is  a pressing  need  to  better  understand  the  impact  from 
the  increase  of  such  services  and  supporting  technology.  To  begin  overcoming  the 
relatively  little  that  is  known  about  how  these  services  are  used,  who  uses  them, 
and  what  impact  these  services  have,  the  statistics  and  performance  measures 
offered  in  this  manual  provide  a start. 

As  reflected  by  the  interest  and  efforts  of  the  many  ARL  libraries  that 
participated  in  the  E-Metrics  project,  the  development  of  library  networked 
statistics  and  performance  measures  continues  to  receive  increased  attention  and 
support.  There  is  broad  recognition  of  the  need  for  network  statistics  and  perfor- 
mance measures  that: 

□ Assist  libraries  in  making  a strong  case  for  support  of  technology  and 
information  infrastructure  by  documenting  their  Internet-based  services 
and  resources; 

□ Assist  libraries  in  demonstrating  the  use  of  digital  collections  in  order  to 
make  a case  for  continued  collection  development  and  support; 

□ Allow  libraries  to  effectively  compare  themselves  to  others  in  terms  of 
Internet-based  collection  and  service  development,  costs,  provision  of 
services,  connectivity,  and  use; 

□ Allow  libraries  to  measure  and  track  internal  changes  to  library  operations 
as  well  as  uses  and  users  of  library  resources  and  services; 

□ Enable  library  directors  and  administrative  library  agencies  to  compete  for 
resources  with  other  organizations  and/or  departments  by  documenting  the 
range,  extent,  and  impact  of  library-provided  networked  services; 

□ Facilitate  the  expansion  from  traditional  library  use  measures  such  as 
circulation,  reference  transactions,  interlibrary  loans,  etc.,  to  include 
network  measures  that  describe  the  nature  and  use  of  library-based  net- 
work activities  and  resources; 

□ Provide  a decision-making  framework  for  library  staff,  managers,  and 
administrators  to  determine  resource  allocation  strategies  and  meet  other 
management  needs; 

□ Provide  a means  through  which  to  measure  the  quality  of  library  services 
and  resources  in  the  networked  environment. 

These  and  other  factors  point  to  the  overall  importance  of  the  development, 
collection,  and  reporting  of  library  network  statistics  and  performance  measures  to 
facilitate  collections  decisions,  cost  analysis,  justification  of  services,  services 
planning  and  evaluation,  and  a host  of  other  activities.  It  is  hoped  that  the  statis- 
tics and  measures  developed  herein  help  fill  many  of  the  needs  faced  by  academic 
and  research  libraries. 

However,  there  are  a number  of  issues  and  challenges  that  affect  the  library's 
ability  to  collect  statistics  and  measures  to  describe  its  electronic  resources  and 
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services.  Some  academic  and  research  libraries  possess  inadequate  resources, 
staffing,  and  expertise  to  collect,  manage,  and  report  the  data  related  to  describing 
networked  services.  For  these  libraries,  some  organizational  development  and 
commitment  to  collecting  and  using  these  data  may  be  necessary  to  take  advantage 
of  the  measurement  tools  and  techniques  outlined  in  this  manual.  The  discussion 
of  measurement  issues  in  Parts  1 through  3 of  the  E-Metrics  Phase  II  Report 
<http://www.arl.org/stats/newmeas/emetrics/>  can  assist  libraries  in  better  under- 
standing why  such  measurement  is  essential. 
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Appendix  A:  List  of  Statistics  Considered 


Categories 

E-Metrics  rl.O  (Initial  list) 

E-Belrics  ill  (Berised  list) 

E-Metrics  t.2.0  (Field  Tesl  list) 

Bfioorcfi 

□ Number  of  electronic  full-text 
journals  (hosted  by  library) 

□ Number  of  electronic  full-text 
journals  (throush  subscription) 

□ Number  of  librarians  provides 
electronic  reference 

□ Number  of  public  access 
workstations 

□ Number  of  electronic  full-text 
periodicals  (hosted  by  library) 

□ Number  of  electronic  full-text 
periodicals  (throush  institutional 
subscription) 

□ Number  of  electronic  full-text 
periodicals  (throush  consortia  and 
other  arransements) 

□ Number  of  electronic  reference 
databases  (throush  institutional 
subscription) 

□ Number  of  electronic  reference 
databases  (throush  consortia  and 
other  arransements) 

□ Number  of  electronic  books 

□ Number  of  staff  providins 
electronic  reference 

□ Number  of  public  access 
workstations 

□ Number  of  electronic  full-text 
journals  (institutional) 

□Number  of  electronic  full-text 
journals  (consortia) 

□ Number  of  electronic  reference 
sources  (institutional) 

□ Number  of  electronic  reference 
sources  (consortia) 

□ Number  of  electronic  books 
(institutional) 

□ Number  of  electronic  books 
(consortia) 

Die 

□ losins  (sessions) 

□ Queries  (searches) 
□Turn-aways  (requests  exceed 
simultaneous  user  limit) 

□ Items  examined  (viewed, 
downloaded,  emailed,  printed) 

□ Total  user  connection  time  to 
vendor  databases 

□Virtual  visits  to  networked 
library  resources 

□ Electronic  reference 
transactions 

□ Number  of  people 
participated  in  user  instruction 
on  electronic  resources 

□ Number  of  losins  (sessions)  to 
networked  library  resources 

□ Electronic  reference  transactions 

□ Number  of  Losins  (sessions)  to 
electronic  databases 

□ Queries  (searches) 

□ Total  connection  time  to 
electronic  databases 

□ Items  examined  (viewed, 
downloaded,  emailed,  printed)  to 
electronic  databases 
□Turn-aways  (requests  exceed 
simultaneous  user  limit) 

□ Number  of  people  participated 
in  user  instruction  on  electronic 
resources  and  services 

□Number  of  electronic  reference 
transactions 

□ Number  of  losins  (sessions)  to 
electronic  databases 

□ Number  of  queries  (searches)  in 
electronic  databases 

□ Items  examined  in  electronic 
databases 

Coit 

□Cost  of  electronic  database 
subscriptions 

□Cost  per  items  examined 
(subscribed  databases) 

□ Cost  of  electronic  files  (one- 
time/monosraphic  purchase) 

□ Cost  of  electronic  full-text 
periodicals  subscriptions 
□Cost  of  electronic  reference 
databases  subscription 

□ Library  contribution  to  consortia 
for  electronic  databases 

□Cost  of  electronic  full-text  journals 
□Cost  of  electronic  reference 
sources 

□Cost  of  electronic  books 

□ Library  expenditures  for  bib. 
utilities,  networks,  and  consortia 

□ External  expenditures  for  bib. 
utilities,  networks,  and  consortia 

Local  Digital 
Collfttioo 

□Cost  of  internal  digital 
collection  construction 

□Cost  of  internal  disital  collection 
construction 

□ Size  of  library  digital  collection 

□ Use  of  library  disital  collection 
□Cost  of  disital  collection 
construction  and  manasement 

Performance 

Heamei 

□ Percentase  of  electronic  . 
reference  transactions  of  total 
reference 

□ Percentase  of  electronic 
materials  use  of  total  library 
materials  use 

□ Percentase  of  remote  library 
visits  of  all  library  visits 

□ Ratio  of  public  access 
workstations  to  university 
population  (number  of  faculty, 
staff,  and  students) 

□ Percentase  of  electronic 
reference  transactions  of  total 
reference 

□ Percentase  of  electronic 
materials  use  of  total  library 
materials  use 

□ Percentase  of  remote  library 
visits  of  all  library  visits 

□ Percentase  of  electronic  titles  to 
all  periodicals 

□ Percentase  of  electronic  books 
to  all  monosraphs 

□ Ratio  of  public  access 
workstations  to  university 
population 

□Cost  per  items  examined  in 
individually  subscribed  databases 

□ Percentase  of  electronic  reference 
transactions  of  total  reference 

□ Percentase  of  electronic  materials 
use  of  total  library  materials  use 

□ Percentase  of  remote  library  visits 
of  all  library  visits 

□ Percentase  of  electronic  books  to 
all  monosraphs 

40 


BEST 


419 


Appendix  B:  Forms 


Forms  List 

Figdre  B.l  Sample  Electronic  Resource  Cost  Report  Form 
Figure  B.2  Sample  Consortia  Expenditure  Report  form 
Figure  B.3  Sample  Consortia  Funding  Report  Form 
Figure  B.4  Sample  Library  Digital  Collection  Inventory  Report  Form 
Figure  B.5  Sample  Digital  Collection  Iitems  Added  Report  Form 
Figure  B.6  Sample  Library  Digital  Collection  Report  Form 
Figure  B.7  Sample  Digital  Collection  Access  Report  Form 
Figure  B.8  Sample  Digital  Collection  Cost  Report  Form  -Personnel 
Figure  B.9  Sample  Digital  Collection  Cost  Report  Form 
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Figure  B.1  (for  C1-C3) 


Sample  Electronic  Resource  Cost  Report  Form 


Reporting  Period: 
Name  of  library: 


Resource/  Consortium  Name  and  Type 

Full-teit  journals  (1) 
Reference  sources  (2) 
Electronic  books  (3) 

Cost 

Comments 

Sub  Total  (1) 

Sub  Total  (2) 

Sub  Total  (3) 

Grand  Total  (1+2+3) 
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Figure  B.2  (for  C4) 


Sample  Consortia  Expenditure  Report  Form 


Reporting  Period: 
Name  of  library: 


Cods  o r 1 i n m N an  e 

A mo u d t 

/'•••  Com  I t 0 1 s 
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Figure  B.3  (for  C5) 


Sample  Consortia  Funding  Report  Form 


Reporting  Period: 
Name  of  library: 


Consortium 

Name 

Total  Funding 
Amount 

Amount 
Attributable 
to  the 
Library 

Comments 
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Figure  B.4  (for  D1) 


Sample  Library  Digital  Collection  Inventory  Report  Form 


Reporting  period: 
Name  of  library: 


ETDs 

• -'V  , 

Visual 

Materials 

Tells 

Audio/?  ideo/Multime- 
’’  ' dia 

'j  Total 

Titles 

(D 

Items 

(2) 

Titles 

(3) 

Titles 

(4) 

Titles 
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Figure  B.5  (for  D1) 


Sample  Digital  Collection  Items  Added  Report  Form 


Reporting  period: 
Name  of  library: 
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Figure  B.6  (for  D1) 


Sample  Library  Digital  Collection  Report  Form 


Reporting  Period: 
Name  of  library: 
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Information  Type  >r. 
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Figure  B.7  (for  D2) 


Sample  Digital  Collection  Access  Report  Form 


Reporting  Period: 
Name  of  library: 


Project 

Name 

Serier 

Name 

Director; 

Location 

Title 

Access 

Count 

Item 

Access 

Count 

Total 

Searches 

Comments 

48 


Figure  B.8  (for  D3) 


Sample  Digital  Collection  Cost  Report  Form  - Personnel 


Reporting  Period 
Name  of  library: 


Raise 

Posilioo 

(Aooual . Salary) 

m 

(Slafr  Costl 

49 
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Figure  B.9  (for  D3) 


Sample  Digital  Collection  Cost  Report  Form 


Reporting  Period: 
Name  of  library: 


Project  Name 

Expense  Type 

Amoant 

Project  (name)  Total 

Project  (name)  Total 

Library  Total 
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Introduction 

It  is  not  enough  simply  to  develop  measures  and  collect  statistics  related  to  library 
networked  resources  and  services.  Indeed,  as  state  legislatures  increasingly  tie  budgets  to 
performance  and  regional  accreditation  boards  begin  emphasizing  the  need  to  articulate 
outcomes,  it  is  important  for  research  libraries  to  decide  what  their  outcomes  should  be  and  to 
determine  how  to  connect  measures  and  statistics  to  these  outcomes  at  both  the  library  and  the 
university  levels. 

As  part  of  the  overall  E-Metrics  project,  the  study  team  has  undertaken  a number  of 
outcomes-related  activities.  During  Phase  II  of  the  E-Metrics  project  (November  2000  - June 
2001),  the  study  team  was  engaged  in  an  extensive,  ongoing  effort  to  develop  evolving 
frameworks  for  understanding  and  graphical  models  for  depicting  measurable  library  inputs 
and  outputs  in  the  context  of  indicators  of  institutional  outcomes.  In  the  third  and  last  phase 
(ending  January  2002),  the  study  team  concluded  its  work  on  outcomes  issues  along  with  the 
other  remaining  components  of  this  project. 

The  study  process  began  with  a review  of  literature  concerning  library  and  institutional 
assessment  to  better  understand  what  other  efforts  had  accomplished,  with  an  eye  toward 
finding  documented  linkages  between  library  outputs  and  institutional  outcomes.  We  found 
that,  while  the  problem  was  clearly  defined  and  its  significance  well  appreciated  and  often 
noted  in  the  literature,  there  has  been  little  work  toward  actually  identifying  linkages  and 
developing  models  that  ARL  member  libraries  could  use  to  determine  how  to  best  measure 
their  impact  on  the  outcomes  of  the  universities  they  support.  Accordingly,  the  study  team 
undertook  laying  a foundation  upon  which  such  efforts  could  succeed  as  research  progresses. 

This  paper  reflects  and  reports  on  the  study’s  efforts  and  findings  to  date;  discusses  our 
current  process  framework  for  approaching  the  issues  and  practices  of  institutional  outcomes 
assessment  and  support  in  academic  and  research  libraries;  and  raises  continuing  questions 
and  issues  that  should  be  considered  at  individual  libraries  and  in  future  research,  particularly 
with  respect  to  the  organization’s  cultural  context  where  outcomes  assessment  occurs. 


Background  and  Methodology 

An  important  component  of  E-Metrics  project  involved  developing  a framework  and 
graphical  model  that  can  ultimately  link  the  study’s  proposed  network  statistics  and  measures 
to:  (1)  educational,  research,  and  service  outcomes  in  higher  education  institutions;  and  (2) 
educational,  research,  and  service  outcomes  in  higher  education  libraries.  In  order  to  engage 
in  this  task  it  was  important  at  the  outset  to  specify  a working  definition  of  the  key  term 
“outcome”  as  it  has  varying  meanings  for  different  users  in  different  contexts.  As  a working 
definition  that  captures  various  related  aspects  of  the  term  “outcome,”  we  have  used  the 
following: 

An  outcome  is  a clearly  identified  result  or  end  product  that 
occurs  as  a consequence  of  individual  or  combined  activities 
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from  units  at  the  institution.  It  is  a preferred  or  desired  state  and 
ideally  clarifies  specific  expectations  of  what  should  be 
products  from  the  institution.  An  institutional  outcome  can  be 
defined  and  measured  in  such  a way  that  evidence  is  available 
to  determine  the  amount  or  degree  to  which  the  outcome  does, 
in  fact,  occur. 

We  found  it  particularly  important  to  distinguish  between  outcomes  of  interest,  desired 
outcomes,  and  actual  outcomes.  Among  the  myriad  if  not  infinite  outcomes  of  the  research 
university  enterprise  (i.e.,  its  results;  that  which  “comes  out”  of  a university  in  a mechanistic 
sense),  outcomes  of  interest  are  those  outcomes — relatively  few  in  number — on  which  a 
particular  university  chooses  to  focus  its  attention  at  a given  time,  taking  into  account  the 
complex,  ever-changing  array  of  relevant,  local  values.  Of  great  significance,  these  outcomes 
must  not  only  be  important  to  the  university’s  leadership  and  constituency,  but  they  must  also 
be  ones  the  university  determines  it  can  impact  and  measure  meaningfully. 

For  the  selected  outcomes  of  interest,  desired  outcomes  are  the  aspirational  levels  of 
achievement  or  production  an  institution  sets  in  advance  to  determine  whether  it  has  attained 
success  at  a future  time  on  some  important  dimension  of  its  operation.  In  other  words,  they 
are  specific  goals  or  quality  standards  for  outcomes  of  interest.  Actual  outcomes  are  the  real 
achievement  or  production  levels  for  an  outcome  of  interest  as  measured  at  a given  time. 

Our  work  in  this  aspect  of  the  E-Metrics  project  began  with  an  extensive  literature  review, 
a summary  of  which  follows  this  section  and  highlights  some  of  the  key  work  identified  in  the 
area  of  outcomes  assessment.  We  also  conducted  a content  analysis  of  selected  ARL  member 
strategic  planning  documents  at  the  library  and  institutional  levels.  This  analysis  revealed  a 
number  of  commonalities  in  the  institutional  goals  of  ARL  members  and  the  manner  in  which 
those  goals  are  devised  and  articulated.  With  the  insights  gained,  in  addition  to  prior  work  in 
this  area  by  the  team,  we  conducted  structured  interviews  in  order  to  solicit  feedback,  identify 
key  work  and  actors  in  the  field,  and  ground  our  framework  development  in  (a)  related  efforts 
already  underway  and  (b)  the  concerns  of  representative  participants. 

Altogether,  these  efforts  substantially  informed  initial  attempts  to  create  an  analytic 
framework  and  depict  an  idealized  institutional  process  focused  on  key  institutional  outcomes 
of  concern  to  university  administrators  that  can  be  impacted  by  libraries.  An  earlier  version 
of  this  paper  was  drafted  to  convey  the  framework  developed  to  date.  It  was  distributed  early 
in  April  2001  to  stimulate  constructive  dialogue  on  the  listserv  for  project  participants  and  at  a 
project  discussion  forum  held  during  the  CN1  Spring  Task  Force  Meeting  on  April  9,  2001. 

With  this  foundation,  the  study  team  refined  its  working  framework  by  then  employing 
other  methods  to  approach  the  complex  issues  at  hand,  including:  (1)  a discussion  forum;  (2) 
site  visits;  (3)  a policy  analysis  of  accreditation  standards;  and  (4)  a survey  of  ARL  deans  and 
directors,  including  their  relevant  strategic  planning  documents.  A summary  of  our  findings 
from  these  efforts  follows  in  a subsequent  section,  and  an  overview  of  all  methods  employed 
is  in  Figure  1. 
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Figure  1 


Metbodo 

ogies  for  Institutional  Outcomes  Framework  Development 

Technique 

Fu  nction/Pu  rpose 

Content  Analysis 

Gather  various  documentation  and  reports  to  review  development  of  current 
outcomes-oriented  requirements,  strategic  plans,  activities,  and  future  directions 
at  a variety  of  ARL  institutions,  including  project  participants. 

Discussion 

Forum 

Explore  key  issues  identified  to  date  regarding  understanding  and  assessment  of 
library  and  institutional  outcomes.  Findings  informed  broader  and  ongoing  data 
collection  activities  and  framework  development. 

Policy  Analysis 

Identification  and  review  of  policy  instruments  adopted  or  under  consideration 
by  accreditation  bodies,  with  a focus  on  institutional  outcomes  assessment 
requirements.  Findings  helped  to  inform  an  understanding  of  regional  and 
national  trends  in  an  area  critical  to  all  ARL  members  and  institutions  of  higher 
education. 

Site  Visits 

Extensive  interactions  and  observations  of  institutions  selected  for  in-depth 
study.  Findings  were  critical  to  contextualizing  and  integrating  insights  gained 
from  individual  case  documents  and  interviews,  as  well  as  for  making 
significant  refinements  to  the  process  framework. 

Structured 
Interviews  with 
Small  Groups 
and  Individuals 

In-depth  exploration  of  outcomes-related  concerns  and  activities  with  key 
informants  at  selected  institutions  and  their  libraries,  including  site  visits. 

Survey  of  ARL 
Membership 

Uniform  collection  of  responses  to  open-ended  questions  regarding  outcomes- 
related  concerns  and  activities  at  academic  ARL  institutions  and  their  libraries. 

Other  projects  within  the  ARL  New  Measures  Initiative  have  pursued  parallel  inquiries 
into  many  of  the  outcomes-related  issues  explored  in  the  E-Metrics  project  (see 
<http://www.arl. org/stats/newmeas/newmeas,html>  for  more  information  on  the  Initiative). 
Of  particular  note,  the  study  team  benefited  from  the  work  done  by  Ken  Smith,  who  is 
investigating  educational  outcomes  for  university  libraries,  and  Doug  Jones,  who  is  studying 
education  and  research  outcomes  for  university  libraries,  (see 

<http://www.arl.org/stats/newmeas/outcomes/heo.html>  for  more  information).  Their  work 
helped  us  better  understand  the  role  electronic  and  networked  services  play  in  these  areas. 
Likewise,  the  work  being  done  to  study  user  perceptions  of  quality  in  the  LibQUAL+ 
program  has  been  of  keen  interest  and  explored  for  possible  connections  (ARL,  2001). 

For  the  same  reason,  the  study  team  investigated  work  done  outside  of  ARL’s  New 
Measures  Initiative.  One  interesting  study  of  the  library’s  impact  on  sponsored  research 
funding  conducted  by  Brinley  Franklin  found  that  “electronic  services  use  supporting 
sponsored  research  generally  mirrored  the  same  level  of  support  exhibited  by  the  general  use 
of  library  materials  and  services  at  almost  all  types  of  libraries”  (Franklin,  2001).  Franklin 
also  found  “a  high  correlation  between  total  research  and  development  funding  at  an 
educational  institution  and  total  library  expenditures  at  research  universities.”  This  work 
suggests  that  electronic  services  use  be  quantified  to  reflect  the  degree  to  which  a library’s 
investment  in  electronic  services  supports  specific  institutional  outcomes. 

Finally,  outcomes-related  issues  have  been  a growing  interest  to  a variety  of  library  and 
information  professionals  lately  as  well  as  university  administrators.  Therefore,  we  have 
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included  a list  of  some  other  outcomes-related  efforts  and  resources — and  Internet  links  where 
appropriate — for  those  interested  in  pursuing  the  topic  further  (see  Appendix  A). 


Selected  Literature 

McClure  and  Lopata  (1996)  found  that  measures  of  teaching,  learning,  and  research  in 
higher  education  were  generally  inadequate,  making  it  more  difficult  to  assess  the  impact  of 
networked  services  in  those  areas.  Library  administrators  “must  rely  on  intuition  and 
anecdotal  information  as  a basis  for  assessing  the  usefulness  and  value  of  a particular  service” 
(p.  3).  Without  appropriate  measures  of  impact,  library  administrators  are  less  able,  or  in 
some  cases  unable,  to  justify  expenditures  for  networked  services.  Additionally,  they  found 
feedback  mechanisms  allowing  administrators  to  improve  networked  services  for  users  are 
weak. 

Pritchard  (1996)  also  calls  attention  to  the  “lack  of  performance  measures  that  make  sense 
across  institutions  and  that  link  library  processes  to  educational  and  research  outcomes”  (p. 
591).  She  notes  that  as  university  administrators  begin  publicly  questioning  the  need  for 
conventional  libraries,  it  is  vital  that  libraries  are  able  to  link  their  information  resources  to  the 
effectiveness  of  academic  programs  (Pritchard,  1996).  One  approach  that  may  be  helpful  in 
doing  this,  she  points  out,  is  Total  Quality  Management,  which  can  be  used  to  improve  the 
processes  of  university  libraries  (p.  590). 

In  Finland,  where  in  the  early  1990s  publicly  funded  institutions  moved  away  from 
traditional  budgeting  to  performance-based  budgeting,  libraries  have  needed  to  produce 
evidence  of  their  contribution  to  university  performance  (Kokkonen,  1996).  “Though  the 
ultimate  result  of  good  library  performance  is  part  of  good  overall  performance  of  the 
university's  teaching  and  research  functions,  it  is  evident  that  the  methods  developed  from  the 
evaluation  of  teaching  and  research  are  not  suitable  for  measuring  library  performance” 
(Kokkonen,  1996). 

In  a report  of  the  America  Association  of  Higher  Education’s  53rd  National  Conference, 
Simoneaux  and  Miller  (1998)  stated  that  the  provosts  in  attendance  generally  saw  the  value  of 
libraries  in  terms  of  supporting  learning  and  information  literacy,  but  that  the  high  cost  of 
information  resources  and  technology  was  a major  concern.  The  ACRL  “Standards  for 
College  Libraries  2000  Edition”  shows  sensitivity  for  this  concern  by  stating  that  outcomes 
assessment  “should  take  into  consideration  libraries'  greater  dependence  on  technology,  their 
increasing  use  of  online  services,  their  growing  responsibility  to  provide  information  literacy 
skills,  their  increasing  reliance  on  consortial  services,  the  possibility  of  dwindling  financial 
resources  for  collection  development,  and  new  ways  in  which  scholarly  information  is 
published  and  distributed”  (ACRL,  2000). 

Smith  (2000),  in  a paper  prepared  for  ARL,  states  that  outcome  assessment  for  the  library 
should  be  treated  like  any  other  academic  department.  “Like  the  Physics  department,  the 
Library  should  be  able  to  contribute  to  the  achievement  of  learning  outcomes  for  various 
academic  programs  across  the  University”  (p.  9).  He  suggests  that  a step  in  the  right  direction 
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might  start  with  the  library  asking  its  partners  in  academic  departments  to  help  it  determine 
how  best  to  support  and  achieve  learning  outcomes  and  then  tailor  the  roles  of  its 
professionals  accordingly. 

The  ACRL  Task  Force  on  Academic  Library  Outcomes  Assessment  was  charged  with, 
among  other  things,  “developing  a philosophical  framework  for  assessing  libraries  in  terms  of 
desired  campus  outcomes”  (ACRL,  1998).  It  viewed  outcomes  as  “the  ways  library  users  are 
changed  as  a result  of  their  contact  with  the  library’s  resources  and  programs”  (ACRL,  1998). 
The  Task  Force  reports  that,  while  libraries  should  be  concerned  with  outcomes,  measurement 
is  difficult  and  that  the  rigor  involved  in  linking  inputs  to  outcomes  will  require  much 
research  (ACRL,  1 998). 

LibQUAL+,  which  adapted  and  built  upon  SERVQUAL  for  the  research  library 
community,  calculates  the  gap  scores  “between  minimum  and  perceived  expectations  and 
desired  and  perceived  expectations”  (Cook  et  al.,  2001).  By  identifying  areas  that  users  say 
are  below  their  minimum  expectation,  libraries  can  begin  to  address  problems  both  of  user 
perception  and  library  quality  (Cook  et  al.,  2001).  While  measuring  outcomes  directly  is 
more  difficult,  an  instrument  like  LibQUAL+  could  be  developed  to  measure  perceived 
outcomes. 

The  work  of  Cullen  and  Calvert  (1995)  examines,  among  other  models,  the  constituency 
satisfaction  model.  The  researchers  developed  a questionnaire  that  asked  stakeholders  to  rate 
the  usefulness  of  99  indicators,  which  were  determined  through  a review  of  the  literature. 
Means  were  taken  and  the  indicators  were  ranked  for  each  constituent  group.  Lindauer  (1998) 
found  that  there  was  overlap  between  the  Cullen  and  Calvert  findings  and  the  impact 
measures  listed  in  McClure  and  Lopata's  1996  manual  and  used  both  in  her  work. 

Lindauer  presents  a useful  framework  for  assessing  the  library  and  its  networked  services 
in  terms  of  institutional  outcomes  (1998).  Her  framework  depicts  the  foundational  role  that 
infrastructure  plays  and  shows  the  importance  of  student  learning  outcomes  in  a teaching 
institution  by  placing  them  above  other  domains.  Meanwhile,  Orr  (1973)  discusses  the  basic 
internal  processes  of  the  library.  These  particular  works  have  been  of  significant  interest  and 
help  to  us  in  this  project  and  is  much  of  the  basis  for  our  graphical  modeling. 

Lindauer  has  also  produced  an  outcomes  assessment  manual  to  “offer  guidance  for 
improving  the  measurement  and  documentation  of  the  impact  of  community  college  library 
and  learning  resources  programs”  (2000,  p.  2).  This  manual  employs  a core  assessment 
method  Lindauer  identifies  as  “ADICAC”:  Align,  Define,  Identify,  Chart/Collect,  Analyze, 
and  Communicate.  The  ADICAC  process,  though,  designed  specifically  for  community 
colleges,  could  be  modified  to  function  as  an  assessment  methodology  for  research  university 
libraries. 

Hemon  and  Dugan  (2002)  describe  outcomes  assessment  in  their  recent  book  Outcomes 
Assessment  in  Your  Library.  A primary  focus  of  the  book  is  the  answer  to  the  question  “How 
are  users  of  our  library  changed  as  a direct  result  of  their  contact  with  our  collections  and 
services?”  (p.  x).  Given  that  focus,  the  book  presents  a variety  of  approaches  for  libraries  to 
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consider  in  planning,  understanding,  and  undertaking  outcomes  assessment  in  their  library 
settings.  The  book  encourages  an  end-user  and  library  focus  and,  thus,  does  not  necessarily 
provide  a means  through  which  academic  libraries  can  link  their  services  and  resources  to  the 
larger  university  institution.  The  tools,  best  practice  examples,  and  other  material  in  the  book 
are  useful,  however,  to  libraries  that  wish  to  consider  developing  a user-based  outcomes 
assessment  approach  for  their  library  services. 

The  literature  reflects  that  underlying  the  operations  of  a research  library  are  the  core 
values  and  goals  of  its  larger  institution.  In  order  to  know  whether  the  values  are  served  and 
the  goals  achieved,  we  cannot  merely  rely  on  counts  of  networked  resource  use.  Rather,  it  is 
critical  that  measurements  in  the  library  context  be  done  strategically  with  an  eye  toward 
fulfillment  of  institutional  goals. 


Findings 

This  section  contains  additional  detailed  sets  of  findings  from  four  major  techniques  of 
data  collection  employed  in  this  study:  a discussion  forum;  the  site  visits;  the  policy  analysis 
and  review  of  accreditation  standards;  and  the  survey  of  ARL  member  deans  and  directors. 
Additional  materials  in  the  appendices  supplement  the  latter  two  subsections. 

Discussion  Forum 


During  the  Coalition  for  Networked  Information  Spring  Conference,  April  2001,  the  study 
team  conducted  a number  of  discussion  groups  and  individual  interviews  regarding  issues 
related  to  institutional  outcomes  and  the  role  of  such  outcomes  in  terms  of  library  planning 
and  evaluation.  The  following  represent  a brief  summary  of  the  key  issues  identified  during 
these  meetings  that  have  ongoing  significance. 

Clarifying  Outcomes  versus  Goals.  Some  considerable  discussion  occurred  regarding  the 
similarities  and  differences  between  the  terms  “institutional  outcomes”  and  “institutional 
goals.”  There  is  a need  to  resolve  these  definitional  issues  and  to  differentiate  the  notion 
of  institutional  outcomes  from  goals  and  objectives. 

Ambiguous/Meaningless  Outcomes  Statements.  A number  of  participants  commented  on 
the  use  of  institutional  outcomes  statements  that  were  essentially  meaningless  since  they 
had  not  been  adequately  defined  and/or  operationalized.  An  example  given  was  that  the 
president  of  one  university  had  stated  her  desired  outcome  is  to  be  the  best  research 
university  on  the  West  Coast.  The  participant  commented  that  such  statements  were  not 
meaningful  because  the  president  did  not  offer  a basis  for  determining  when  the  university 
would  in  fact  satisfactorily  achieve  this  outcome. 

One-to-One  Linkages  between  the  Library  and  Institutional  Outcomes.  There  was 
considerable  skepticism  on  the  part  of  a number  of  participants  that  specific  individual 
activities  of  the  library  could  be  shown  to  be  a major  or  direct  link  to  a specific 
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institutional  outcome.  Some  participants  thought  that  for  many  instances  the  library’s  role 
would  be  more  indirect  or  as  a supporting  service  that  contributed  to  institutional 
outcomes  in  very  “roundabout”  ways. 

Concern  about  Linking  Library  Inputs  to  Institutional  Outcomes.  One  participant  noted 
that  she  worked  at  a university  library  with  a relatively  small  (compared  to  other  ARL 
libraries)  collection  count/size.  Nevertheless,  the  university  was  ranked  in  the  top  20  of 
all  institutions  receiving  federally  funded  research  grants.  She  stated  (and  others  agreed) 
that  direct  relationships  between  library  input  and  process  measures  and  institutional 
outcomes  could  “backfire”  on  library  administrators  if  such  relationships  were  inverse. 
Arguments  that  more  library  inputs  (however  defined)  would  naturally  improve  the 
accomplishment  of  institutional  outcomes  should  be  carefully  considered  before  being  put 
forth  to  university  administration. 

Comparisons  and  Peers.  There  was  some  discussion  that  comparing  a specific  ARL 
institution  in  terms  of  accomplishing  certain  outcomes  may  not  make  sense  unless  there  is 
an  agreed-upon  peer  group.  Simply  being  an  ARL  institution  does  not,  in  and  of  itself, 
provide  enough  homogeneity  to  make  such  comparisons.  Furthermore,  there  is  an 
inherent  desire  on  the  part  of  members  of  the  Association  of  American  Universities 
(AAU)  to  compare  themselves  (positively)  to  other  institutions.  The  following  concern 
was  raised:  Will  institutional  outcomes  become  another  “tool”  to  compare  one  university 
to  another — oftentimes  with  dysfunctional  results? 

Importance  of  Process  in  Setting  Institutional  Outcomes.  There  was  a general  sense  that 
ARL  libraries  needed  to  develop  a process  to  identify  and  operationalize  library  outcomes 
that  contribute  to  institutional  outcomes.  The  library  is  critical  in  informing  the  university 
of  “appropriate”  institutional  outcomes  to  which  the  library  contributes.  Setting  such  a 
process  is  an  important  method  for  informing  key  stakeholders  in  the  university  of  both 
the  library’s  role  in  institutional  outcomes  and  insuring  that  the  institutional  outcomes  to 
which  the  library  has  (or  may  have)  links  are  appropriate.  Some  participants  thought  that 
the  study  team  might  make  an  important  contribution  to  the  project  by  suggesting  how 
such  a process  might  be  developed. 

Importance  of  Situational/Institutional  Factors.  There  was  wide  agreement  that  a useful 
outcomes  model  ultimately  needs  to  recognize  the  wide  range  of  situational  and 
institutional  factors  that  might  affect  the  role  of  the  library  in  the  development  of  and 
impact  on  institutional  outcomes  at  a particular  campus.  Situational  and  institutional 
factors  that  may  be  relevant  include: 

• Different  missions  of  the  university,  including  missions  that  might  evolve  from 
state  legislatures  and  other  external  factors; 

• Process  by  which  resources  are  allocated  across  campus; 

• Importance  placed  on  institutional  outcomes  and  how  they  are  defined  at  a 
particular  institution; 

• Culture  and  traditions  for  roles  and  responsibilities  of  faculty,  etc.; 
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• The  physical  location  of  the  university,  especially  in  terms  of  a urban  versus  non- 
urban  setting; 

• Organizational  structure  and  governance  structures  of  both  the  library  and  the 
larger  institution; 

• Planning  process; 

• Role  and  extent  of  branch  and/or  departmental  libraries  on  campus; 

• Faculty  involvement  in  library  governance  through  a faculty  senate  library 
committee,  etc.; 

• Technological  infrastructure  and  management  of  that  infrastructure;  and 

• “Responsibility-based  budgeting”  of  units  in  which  each  unit  accounts  for  its  own 
income  versus  expenditures. 


Clearly  there  are  likely  to  be  a significant  number  of  additional  situational  and 
institutional  factors  that  would  vary  from  university  to  university.  The  degree  to  which 
any  model  can  take  into  consideration  the  range  of  situational  and  institutional  factors  may 
be  problematic. 


Point-ofrView  Toward  Outcomes  from  Different  Stakeholder  Groups.  What  might 
constitute  institutional  outcomes,  or  appropriate  institutional  outcomes,  will  vary  from 
one  group  to  another  (deans,  faculty,  trustees,  students,  etc.).  It  could  be  that  the 
outcomes  from  the  library  (as  agreed-upon  by  library  staff  and  administration)  may  or 
may  not  be  those  seen  by  university  administration  as  important  or  appropriate.  Is  there 
some  type  of  hierarchy  of  outcomes  from  different  units  in  the  institution  as  opposed  to 
the  institution  overall?  The  trick  to  institutional  outcomes  is  that  they  should  be  end 
results  that  all  units  of  the  institution  have  a stake  in  achieving — not  just  the  library. 


If/Then  Models  to  Depict  Libraries  and  Institutional  Outcomes.  There  was  some  support 
for  developing  multiple  models  for  the  depicting  the  relationships  between  libraries  and 
institutional  outcomes.  For  example,  IF  the  library  had  certain  situational  and  institutional 
factors,  THEN  the  model  linking  activities  to  institutional  outcomes  would  be  different  for 
a library  in  that  situation  than  one  that  had  differing  situational  and  institutional  factors. 

To  pursue  this  possibility  would  require  a better  understanding  and  operational  description 
of  specific  situational  and  institutional  factors  that,  perhaps  as  a menu  of  possible  factors, 
could  affect  the  way  in  which  the  library  is  associated  with  institutional  outcomes. 


A Menu  of  Possible  Outcomes . Some  participants  thought  that  it  would  be  useful  to 
identify  a menu  of  possible  institutional  outcomes  from  which  an  institution  might  select 
those  outcomes  of  interest  or  importance  to  them.  For  example,  one  outcome  is  “high 
research  productivity”  and  could  be  measured  by  the  institution’s  ranking  of  amount  of 
federally  funded  research  obtained  per  year.  Libraries  (and  institutions)  could  then  select 
the  outcomes  that  were  most  appropriate  for  their  particular  situations.  An  interesting 
example  of  how  one  institution,  Ohio  State  University,  has  articulated  its  goals,  outcomes 
and  indicators  of  achievement  is  in  that  university’s  “Strategic  Indicators  - 2001 : 
Executive  Summary”  (available  at  <http://www.rpia.ohio- 
state.edu/Strategic  analysis/strategic  indicators/2001  /Exec  Summary. pdf>V 
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Joint  Activities/Roles  between  the  Library  and  other  Institutional  Units.  Participants 
believed  that  oftentimes  direct  linkages  from  the  library  to  a specific  institutional  outcome 
might  be  difficult  to  identify  and  measure.  Some  suggested  the  possibility  of  identifying 
groups  of  institutional  units  that  might  all  be  working  toward  a specific  institutional 
outcome.  Such  might  be  a better  approach  for  the  library  to  be  linked  to  outcomes  rather 
than  considering  a library  activity  as  a single  factor  in  producing  an  outcome. 

The  issues  briefly  described  in  this  section  are  not  intended  to  be  a comprehensive  listing 
of  those  identified  and  discussed  by  the  various  participants  in  the  meetings.  They  do, 
however,  suggest  the  extent  to  which  (a)  sufficient  understanding  of  institutional  outcomes, 
(b)  the  relationship  of  institutional  outcomes  to  library  activities  and  outcomes,  and  (c)  how 
an  outcomes  focus  should  be  used  in  planning/evaluation  of  library  services,  requires  much 
additional  thought  and  research. 

Site  Visits 


During  the  week  of  May  21,  2001,  two  members  of  the  E-Metrics  project  team  conducted 
site  visits  at  the  University  of  Arizona  (UA)  and  Arizona  State  University  (ASU),  particularly 
at  their  main  libraries.  Exploring  issues  from  both  institutional  and  library  perspectives,  the 
visits  focused  on  better  understanding  the  manner  in  which  each  university:  (a)  sets  its 
strategic  direction  with  both  institutional  and  library  outcomes  of  interest;  (b)  articulates  those 
outcomes  and  the  results  desired;  (c)  assesses  whether  the  desired  outcomes  have  been 
achieved;  and  (d)  uses  outcomes-related  concerns  and  data  in  decisionmaking. 

On  both  campuses  the  team  conducted  a number  of  in-depth  interviews  and  group 
discussions,  with  both  a variety  of  line  and  administrative  librarians,  as  well  as  with  senior 
university  planning  officers.  In  addition,  they  were  able  to  review  and  discuss  a wide  range  of 
relevant  documents,  many  of  which  were  collected  for  further  review  and  analysis.  Finally, 
they  benefited  from  observing  library  activities  at  both  locations. 

The  study  team  gained  great  insights  into  both  institutions  from  very  helpful  informants 
who  were  without  exception  extremely  generous  with  their  time  and  contributions.  Moreover, 
it  provided  an  opportunity  to  update  members  of  both  universities  with  project  developments, 
and  involve  them  intimately  in  this  component  of  the  project.  Significant  interest  and 
enthusiasm  were  expressed  throughout  the  week  in  this  study’s  focus  on  outcomes,  with  a 
strong  hope  that  this  investigation  continue  to  advance  the  issue  of  a research  library's  role  in 
university  outcomes  by  further  framing  the  discussion  and  pursuing  the  inquiry. 

The  study  of  these  institutions,  separately  or  together,  was  not  an  end  in  itself,  and  neither 
was  in  any  way  evaluated.  Rather,  both  were  selected  as  two  patently  different  organizations 
within  the  same  state  university  system,  providing  an  unusual  opportunity  for  making 
comparisons  and  identifying  contrasts. 

The  University  of  Arizona  Library  is  well  known  for  having  undergone  a major 
reengineering  nearly  a decade  ago  by  implementing  an  extremely  flat  organization  managed 
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almost  entirely  by  teams.  While  the  teams  themselves  are  arranged  in  a structure  with  some 
formalities  that  have  evolved  (and  continue  to  evolve),  and  the  dean  operates  within  the  larger 
structure  of  the  university,  every  individual  staff  member  has  the  opportunity  and  the 
expectation  to  participate — at  one  time  or  another — in  virtually  every  facet  of  library 
operations.  One  of  the  hallmarks  of  this  approach  is  the  overt  empowerment  of  each 
individual  with  corresponding  responsibilities,  and  mutual  reliance  on  everyone  else 
exercising  his  or  her  team-based  power  responsibly.  What  is  more,  everyone  interviewed  or 
observed — regardless  of  regular  job  duties  or  position — was  mindful  of  and  conversant  with 
the  full  array  of  the  library’s  strategic  and  operational  issues,  including  budget  details,  short- 
term objectives,  and  long-term  goals. 

As  with  all  human  endeavors,  there  are  always  inefficiencies  and  uncertainties.  Along 
with  many  instances  of  routine  progress  made  through  unusually  egalitarian  methods,  the 
team  also  observed  a fair  amount  of  time  expended  on  inter-  and  intra-team  monitoring  and 
adjustments,  orienting  and  reorienting.  This  was  not  necessarily  counterproductive,  but  it 
appeared  that  the  absence  of  explicit  goals,  directions,  and  standards  from  the  top  down 
resulted  in  much  confusion  from  the  need  to  generate  such  from  the  bottom  up.  By  and  large 
the  uncertainties  and  confusion  were  viewed  as  an  investment  in  a continuously  learning 
organization  where  every  individual,  ideally,  is  highly  knowledgeable  and  focused  on  quality 
customer  service  and  the  organization’s  greater  good  (i.e.,  the  university’s  outcomes  of 
interest). 

The  organizational  structure  of  the  Arizona  State  University  Libraries  is  relatively 
traditional  with  a hierarchy  of  administration  and  departments  and  line  staff;  and  yet  there  are 
teams  within  the  more  traditional  structure  and  other  instances  of  flexibility  or 
experimentation  as  situations  warrant  or  seem  worth  exploring.  In  contrast  with  the  fluid  UA 
Library,  the  ASU  Libraries  seem  mechanical  with  a top-down  management  system.  But  this 
is  not  a pejorative  characterization  as  a general  matter,  and  the  observations  of  ASU’s  more 
familiar  operation  mode  did  not  suggest  anything  other  than  a well-run  system. 

While  the  traditional  hierarchical  system  has  become  associated  in  the  minds  of  many 
with  the  negative  connotations  of  bureaucracy,  that  is  a misleading  image  and  disguises 
significant  efficiencies  when  the  system  is  well  managed.  A more  rigid  structure  with  well- 
established  and  clearly  communicated  goals,  directions,  and  standards  can  be  quite 
empowering  for  employees  at  any  level,  albeit  in  a different  sense  from  that  of  a flat 
organization.  Line  employees  can  focus  on  their  particular  jobs  and  not  on  being  an  integral 
part  of  managing  the  organization.  And  that  appeared  to  be  the  case  at  ASU.  Moreover,  the 
organization’s  managers  are  mindful  of  the  university’s  mission,  values,  and  outcomes  of 
interest,  which  are  conveyed  throughout  the  library  even  if  every  line  employee  is  not 
conscious  of  it  in  the  same  manner  as  those  at  the  UA  Library. 

The  key  insight  regarding  the  outcomes  framework  development  that  arose  out  of  the  site 
visits  was  that,  notwithstanding  the  obvious — and  in  ways  profound — differences,  both 
universities  invest  a very  high  importance  in  the  library  deans.  The  deans’  role  and  the  way 
they  exercise  it  is  not  to  the  exclusion  of  the  importance  of  other  library  staff,  for  all  interact 
in  one  way  or  another  with  various  campus  customers  and  stakeholders,  and  the  staff  are  at 
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the  heart  of  the  day-to-day  operations.  But  the  deans  are  the  public  and  official  champions  of 
the  library  on  campus  among  other  deans,  the  provost,  and  the  president. 

The  administrators  at  both  universities  apparently  convey  to  the  library  an  appreciation  (or 
assumption)  of  the  library’s  role  in  contributing  to  the  larger  institutional  values,  and  rely  on 
the  library  deans  to  present  the  achievements  and  contributions  of  their  libraries  in  a manner 
the  deans  deem  appropriate.  Thus,  our  understanding  is  that  in  both  cases  the  library  deans 
are  free  to  present  the  library  to  senior  administrators  and  peers  in  whatever  manner  argues 
best  for  what  the  library  does  to  contribute  to  its  university,  without  any  imposed 
requirements  as  to  substance  (although  the  ASU  administration  has  directed  via  memorandum 
all  departments,  including  the  library,  to  tie  budget  requests  to  the  university’s  strategic  plan). 

This  is  highly  significant:  at  present  these  two  very  different  library  organizations  are 
similarly  situated  vis-a-vis  presentation  of  the  library  as  a whole  to  the  academic  community 
it  serves.  The  deans  and  associated  staff  must  define  the  library — and  redefine  it  continually 
as  their  local  and  larger  environment  evolves;  they  must  make  their  contributions  to  their 
respective  communities,  ultimately  in  terms  of  current  outcomes  of  interest;  and  they  must 
demonstrate  how  they  are  actually  contributing  to  the  achievement  of  desired  outcomes.  But 
in  the  absence  of  internally  imposed  measures,  and  in  the  absence  of  widely  recognized 
measures  that  supports  such  a demonstration,  clear  linkages  between  library  resources  and 
services — electronic,  networked,  or  otherwise — and  the  institutional  outcomes  they  must 
ultimately  support,  remain  unspecified.  ASU  and  UA  share  this  situation.  They  also  seem  to 
share  the  same  solution:  excellent  communication  of  what  the  library  does. 

Accreditation  Review 


The  study  team  engaged  the  services  of  Bonnie  Gratch-Lindauer  to  follow  up  on  one 
aspect  of  her  previous  work  (Lindauer,  1998)  and  conduct  a review  of  relevant  accreditation 
standards  as  they  relate  to  outcomes  and  networked  services  (see  Appendix  B for  background 
on  the  methodology  employed  and  the  documents  reviewed).  For  her  review,  the  major  focus 
was  to  help  illustrate:  (1)  the  need  for  research  university  libraries  to  demonstrate  the 
outcomes  of  electronic  and  networked  services;  (2)  the  need  for  such  libraries  to  demonstrate 
any  outcomes  apart  from  electronic  and  networked  services;  and  (3)  the  need  for  research 
universities  to  show  the  connection  between  the  use  of  electronic  and  networked  services  and 
the  fulfillment  of  their  missions/goals. 

With  regard  to  the  need  for  research  university  libraries  to  demonstrate  the  outcomes  of 
electronic  and  networked  services,  the  standards  and  supporting  documents  contain  few 
references  to  electronic  and/or  networked  resources  and  services.  Only  five  instances  of  the 
terms  network  and/or  electronic  resources  and  services — or  similar  terms  such  as  electronic 
bibliographic  databases  or  online  catalogs — were  found.  However,  relatively  general  but 
related  terms  like  information  resources  and  services,  information  technology  resources  and 
services,  or  information  technologies,  were  more  commonly  used  in  context.  This  broader 
terminology  clearly  includes  electronic  and  networked  resources  and  services,  and  its 
increasing  importance  in  accreditation  standards  underscores  the  growing  expectation  that 
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academic  libraries  demonstrate  how  these  resources  and  services  support  student  learning  and 
faculty  research. 

Regarding  the  need  for  libraries  to  demonstrate  any  outcomes  apart  from  electronic  and 
networked  services,  the  emphasis  on  assessing  student  learning  and  other  outcomes  is 
generally  stronger  in  those  standards  that  have  been  or  are  currently  undergoing  revision  since 
1998.  Moreover,  as  the  Middle  States  Commission  on  Higher  Education  detailed,  Title  IV  of 
the  1998  Higher  Education  Amendments  requires  universities  receiving  federal  monies  to 
have  an  outcomes  assessment  plan  that  includes  “a  review  of  the  institution's  success  with 
respect  to  student  achievement  in  relation  to  mission.  Institutions  should  include  in  the  self- 
study  a review  of  course  completion,  graduation  rates,  state  licensure  exam  pass  rates,  and 
other  data  as  appropriate  to  the  mission  of  the  institution  and  the  programs  it  offers”  (Middle 
States,  2000,  p.  32). 

As  for  the  need  to  show  the  connection  between  the  use  of  electronic  and  networked 
services  and  the  fulfillment  of  missions/goals,  the  findings  were  that,  while  there  may  not  be  a 
need  to  demonstrate  the  outcomes  of  electronic  and  networked  resources  and  services,  there  is 
a need  to  provide  evidence  that  documents  the  connections  between  their  use  and  fulfillment 
of  the  institution’s  goals.  Four  of  the  documents  reviewed  support  this.  Lindauer  also  found 
that  university  libraries  need  to  make  more  explicit  and  public  the  connections  between  the 
following: 

a. )  How  their  resources  and  services  support  institutional  goals  (in  some  of  the 

standards,  appraisal  of  annual  institutional  goals  and  progress  in  their 
accomplishment  is  suggested  as  a type  of  evidence  contributing  to  institutional 
effectiveness); 

b. )  How  their  resources  and  services  are  used,  by  whom,  and  the  effects  or  impacts  of 

this  use;  and 

c. )  How  their  strategic  plans  and  assessment  plans  support  the  institution’s  planning 

documents  and  assessment  process  and  how  the  findings  of  the  library’s 
assessment  activities  contribute  to  the  achievement  of  the  institution’s  mission  and 
goals. 

The  key  observations  from  the  review  are  as  follows: 

1 . The  majority  of  the  outcomes  and  outcomes-related  statements  that  refer  to 
libraries  and  information  resources  are  located  in  sections  of  the  standards  that 
deal  with  the  education  program  and  institutional  effectiveness. 

2.  The  use  of  library  and  information  resources  is  connected  to  student  learning 
outcomes  in  four  of  the  documents,  and  evidence,  such  as  inclusion  in  course 
syllabi  and  integration  of  library  use  into  the  undergraduate  curriculum,  are  offered 
as  measurable  indicators  for  assessment  purposes  in  two  of  the  documents. 
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3.  The  university  library’s  role  in  helping  students  develop  information  literacy  skills 
is  an  important  student  learning  outcome  referenced  in  four  of  the  documents  and 
in  the  “Best  Practices  for  Electronically  Offered  Degree  and  Certificate  Programs,” 
endorsed  by  the  accrediting  commissions. 

4.  Assessing  student  needs,  perceptions  and  levels  of  satisfaction  with  educational 
support  services  (i.e.,  library  and  information  services)  and  demonstrating  that  the 
findings  from  these  user  studies  are  used  for  program  improvement  are 
fundamental  expectations  of  all  the  regional  accrediting  commissions. 

5.  Appraisal  of  annual  institutional  goals  and  progress  in  their  accomplishment  is 
suggested  as  a type  of  evidence  contributing  to  institutional  outcomes,  or  in  some 
of  the  documents  the  phrase  used  is  “institutional  effectiveness.” 

6.  All  of  the  standards  describe  the  need  for  institutions  to  have  an  assessment  or 
evaluation  plan  and  to  document  that  the  findings  are  utilized  for  program 
improvement.  Some  of  the  documents  clarify  this  requirement  to  mean  that  each 
program  or  unit  should  have  an  assessment  plan. 

7.  Several  of  the  documents  refer  to  the  campus  climate  or  the  institutional 
environment  that  supports  teaching  and  learning.  Three  specifically  connect 
library  and  information  resources  and  services  to  the  quality  of  the  learning 
environment.  The  implication  is  that  university  libraries  should  clearly  describe 
what  resources  and  services  they  provide  that  directly  support  the  learning 
environment,  how  these  are  used  and  with  what  effects  on  students  and  faculty. 


The  role  of  accrediting  bodies  is  widely  acknowledged  but  its  import  cannot  be  overstated. 
First,  there  is  the  obvious  matter  of  gaining  or  maintaining  accreditation  with  a regional 
commission,  as  the  failure  to  do  so  would  supersede  all  other  organizational  concerns. 
Secondly,  their  increasing  focus  on  outcomes  assessment  could  be  a bellwether  for  more 
requirements  of  systematic  outcomes  measurement  and  outcomes-based  justification  from  a 
range  of  stakeholders,  including  universities  and  state  legislatures. 

Survey  of  ARL  Members 

As  part  of  the  data  collection  for  the  outcomes  portion  of  the  E-Metrics  project,  a Web 
survey  was  posted  on  the  ARL  website  in  June  2001,  with  a request  sent  to  the  ARL 
membership  asking  for  academic  members’  responses  to  six  open-ended  questions;  the  survey 
also  solicited  any  supporting  or  amplifying  documents  relevant  to  the  issues  raised  (see 
Appendix  C for  the  survey  instrument).  Given  the  nature  of  the  survey,  the  study  team  was 
pleased  to  receive  nineteen  excellent  responses,  many  including  documents  sent  separately  or 
by  identifying  appropriate  URLs  on  the  Web.  The  responses  represented  a cross-section  of 
the  membership,  revealing  a variety  of  insights,  circumstances  and  perspectives  that 
underscore  the  importance  of  this  inquiry  as  well  as  its  complexity. 

The  following  two  lists  contain  a series  of  insights  and  conclusions  drawn  from  the  survey 
as  a whole.  The  first  list  pertains  to  the  university  level,  while  the  second  focuses  on  the 
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university’s  library.  (A  discussion  of  selected  responses  to  each  question  may  be  found  in 
Appendix  D.) 

Regarding  the  university: 

• University  mission  statements  tend  to  be  rather  similar.  All  such  statements 
submitted  for  this  survey  contain  education,  research,  and  service  components. 

• Most  university  strategic  plans  make  some  mention  of  the  library,  particularly  with 
regard  to  information  technology  and  electronic  resources  in  support  of  teaching, 
research,  and  learning. 

• Several  universities  have  goals  statements  that  either  apply  to,  or  make  mention  of, 
the  library,  especially  regarding  increasing  information  literacy  of  students  and 
supporting  academic  programs  and  faculty  research. 

• The  documents  reviewed,  for  the  most  part,  do  not  contain  outcome  statements  per 
se,  but  typically  state  a vision  for  the  university  and  goals  that  describe  an  ideal  or 
future  state  of  the  university. 

• No  document  reviewed  mentioned  specific  ways  to  assess  programs  or  units  on 
campus.  It  may  be  that  this  kind  of  information  would  be  found  in  other  university 
documents  or  that  assessment  tends  not  to  be  a high  priority  on  campus,  at  least 
not  at  the  detail  level  in  the  responding  institutions. 

• At  universities  where  money  is  especially  tight,  the  university  seems  to  be  more 
focused  on  budget  justification.  At  such  a university  the  library  is  typically 
expected  to  demonstrate  how  it  helps  the  university  toward  its  mission. 

• Communication  from  the  provost  (or  other  senior  administrators)  articulating 
specific  goals  to  the  library  seems  to  be  vital  in  maintaining  a library  that  can  meet 
the  needs  of  the  campus.  Respondents  who  stated  that  goals  were  more  vague  or 
not  formally  articulated  seem  to  be  having  the  biggest  struggles  with  stating  their 
impacts  and  getting  bigger  budgets. 

Regarding  the  library: 

• Almost  every  library  mission  statement  included  in  this  analysis  stated  that  the 
library  existed  to  support  the  teaching,  research,  and  service  mission  of  the 
university. 

• Many  libraries’  vision  statements  expressed  an  increase  in  the  amount  of 
information  technology  and  electronic  resources  owned  and  used  by  the  library, 
and  asserted  that  this  would  help  them  better  serve  customers. 


January  2002 


446 


15 


ARL  E-Metrics  Project 


Toward  a Framework  of  Outcomes 


• As  one  library  pointed  out,  environment  is  the  most  variable  factor  in  strategic 
planning.  Libraries  must  know  their  current  environment  and  able  to  plan  for  their 
future  environment. 

• Many  libraries  started  creating  strategic  plans  in  only  the  last  few  years,  but  find 
them  vital  for  guiding  the  work  they  do. 

• Many  libraries  are  required  to  support  the  goals  of  the  university  and  use  that  as  a 
starting  point  for  setting  their  goals. 

• Few  libraries  have  outcomes  explicitly  stated  in  their  strategic  plans.  Many, 
however,  do  articulate  desired  states  for  the  library. 

• Libraries  have  to  be  selective  in  the  resources  and  services  they  offer  because 
budgets  do  not  allow  them  to  do  and  buy  everything  desired.  Some  libraries  noted 
that  budgets  are  so  low  that  they  have  had  to  cut  vital  resources  (like  journals)  and 
services.  These  libraries  see  creating  a strong  digital  collection  as  a way  to  offer 
more  access  and  to  share  resources  with  other  libraries. 

• Libraries  are  doing  many  things  to  support  other  units  on  campus.  They  work  to 
make  sure  that  programs  have  the  necessary  information  resources,  work  with  new 
programs  to  get  resources  in  place,  go  into  classrooms  to  teach  information 
literacy/research  skills,  and  help  faculty  create  digital  collections. 

Many  of  these  conclusions  concern  similar  topics  also  identified  in  the  discussion  forum 
findings. 


Discussion 

Lindauer’s  study  (1998)  particularly  informed  earlier  versions  of  a graphical  model  that 
was  distributed  for  discussion  in  a preliminary  version  of  this  paper  at  the  2001  CN1  Spring 
Task  Force  Meeting  of  the  ARL  E-Metrics  project  participants.  In  her  articles,  Lindauer 
describes  five  assessment  domains  for  libraries  and  connects  goals  of  the  university  with 
activities  and  measures  within  the  library.  Assessment  domains  for  the  teaching-learning 
library  that  she  listed  in  her  article  include:  (1)  learning  outcomes  and  enabling  instructional 
outputs;  (2)  faculty/academic  staff  teaching  effectiveness,  scholarly  productivity,  and 
professional  development;  (3)  institutional  viability  and  vitality;  (4)  access,  availability,  and 
use  of  teaching-learning  recourses;  and  (5)  infrastructure — human  resources,  collections,  and 
equipment/facilities  (see  Figure  2 below,  from  Lindauer,  1998,  p.  557). 
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Figure  2.  Assessment  Domains  for  the  Teaching-Learning  Library 


Learning  Outcomes  and  Enabling  Instructional  Outputs 


Faculty/Academic  Staff  Teaching  Effectiveness,  Scholarly 
Productivity,  and  Professional  Development 


Institutional  Viability  and  Vitality 


Access,  Availability,  and  Use  of  Teaching-Learning  Resources 


Infrastructure:  Human  Resources,  Collections,  and  Equipment/Facilities 


Our  initial  graphical  model  began  to  depict  the  process  by  which  an  academic  research 
library  helps  meet  the  goals  of  particular  departments  and  functional  units  within  a university, 
which  in  turn  contribute  to  institutional  goals,  while  acknowledging  that  libraries  may  also 
contribute  more  directly  (see  Figure  3,  where  “Lindauer  2,”  for  example,  refers  to  the  second 
of  Lindauer’s  assessment  domains,  counting  from  the  top  down  in  Figure  2).  In  our  earlier 
model,  this  idealized  process  begins  with  the  university  administration,  which,  from  a 
standpoint  of  maintaining  viability,  vitality  and  core  values,  conceives  goals  that  are 
articulated  to  the  institution’s  various  academic  and  support  units.  From  these  goals  and  the 
needs  of  the  various  units,  decisions  about  budget  allocations  are  made.  The  library  receives 
funding,  gains  awareness  of  the  needs  of  the  university  generally  and  of  other  campus  units, 
and  makes  decisions  about  which  resources  and  services  to  purchase  and  offer  its  patrons. 

We  hypothesized  that  there  is  an  explicit  or  otherwise  tacit  expectation  on  the  part  of  the 
library  and  university  administration  that — in  the  absence  of  feedback  to  the  contrary — actual 
investments  in  the  library,  and  customer  use  of  library  resources  and  services,  match  with  the 
values  and  goals  of  the  institution,  and  meet  the  needs  of  other  academic  units.  In  the  ideal 
case,  beneficial  effects  result,  other  units  are  supported,  and  the  goals  of  the  institution  are 
achieved.  Outcomes  at  the  institutional  level  are  measured  against  values  and  goals  and  help 
shape  future  goal  setting  and  budgeting  decisions. 
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FIGURE  3.  Initial  Graphical  Model  of  the  Outcomes  Assessment  Process 
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Based  on  substantial  constructive  input  from  many  participants  in  this  project,  the  current 
graphical  depiction  of  the  outcomes  assessment  process  is  a refinement  of  the  same 
underlying  framework  (see  Figure  4).  Although  it  does  not  show  how  particular  services  and 
resources  contribute  to  outcomes,  the  model  does  describe  how  a library  dean  or  director  can 
begin  thinking  about  targeting  library  services  toward  the  outcomes  of  interest  to  the 
university,  and  measure  their  contributions  and  impacts  more  effectively  in  actual  outcomes. 
Libraries’  contributions  to  actual  university  outcomes  will  typically  be  indirect  and/or  partial; 
thus,  actual  outcomes  at  the  university  level  will  not  necessarily  give  a clear  indication  of 
achievement  (or  not)  of  success  from  the  library  level.  Linking  measures  to  outcomes  cannot 
be  done  without  first  considering  what  outcomes  are  desired  and  what  effects  achieving  those 
outcomes  could  have.  Library  personnel  can  then  explore  surrogate  and/or  composite 
measures  that  are  accurate  and  reliable  indicators  of  actual  outcomes  of  interest. 


FIGURE  4.  Current  Graphical  Model  of  the  Outcomes  Assessment  Process 
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University  research  libraries  are  established  to  support  the  broad  research,  education,  and 
service  goals  that  are  fundamental  to  the  mission  of  the  institutions  they  serve.  Beyond 
helping  to  fulfill  the  university  mission,  a research  library  must  be  able  to  help  the  larger 
institution  reach  its  more  concrete  but  shifting  goals.  These  goals  may  be  articulated  in 
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strategic  planning  documents,  in  conversations  with  the  provost  or  academic  deans,  in 
fulfilling  regional  accreditation  standards,  or  in  state  legislation. 

It  is  important  to  be  highly  aware  of  these  goals  and  to  be  able  to  target  library  resources, 
services,  and  programs  to  help  meet  institutional  goals.  Doing  so  is  critical  to  the  university 
research  library  being  seen  as  a vital,  contributing  part  of  the  university.  To  address  these 
goals  and  measure  them  effectively,  it  is  important  to  ask  three  key  questions: 

1.  What  is  the  desired  operational  state  of  the  university? 

2.  How  can  the  library  help  the  university  achieve  this  state? 

3.  How  will  the  library  know  when  it  has  been  successful  in  helping  the  library 
achieve  this  state? 

The  first  question  helps  the  library  better  understand  its  operating  environment.  Knowing 
what  the  university  wants  to  be  now  and  in  the  future  helps  library  administrators  understand 
what  customers  need. 

The  second  question  helps  in  making  decisions  about  which  of  the  many  goals  of  the 
university  are  helpful  for  the  library  to  focus  on.  In  addition,  answering  this  question  helps 
the  library  make  decisions  about  what  action  it  must  take  to  contribute  to  the  fulfillment  of 
those  goals.  Existing  services  and  programs  may  address  them,  programs  may  have  to  be 
fine-tuned  and  resources  upgraded,  or  new  programs  and  resources  may  need  to  be  developed 
and  acquired. 

The  third  question  helps  the  library  craft  the  measures  that  will  provide  indications  of 
success.  This  is  extremely  important  and  must  be  thought  about  carefully  because  direct 
outcomes  measurement  is  often  difficult  or  impossible.  It  may  be  necessary  to  develop 
several  measures  that  work  together  to  indicate  an  outcome  state  or  to  use  surrogate  measures 
such  as  perception  surveys.  One  framework  for  developing  such  measures  is  described 
below. 

The  order  in  which  these  questions  are  addressed  is  also  vital.  Before  measurements  can 
be  derived,  library  administrators  must  know  what  they  want  to  measure  and  how  the  library’s 
impact  is  likely  to  occur.  Without  this  perspective  and  focus,  outcomes  measurement  cannot 
be  targeted  and  loses  meaning.  In  other  words,  it  is  very  difficult  to  gain  useful  insights  about 
outcomes  when  measures  are  not  designed  with  outcomes  explicitly  in  mind. 

ARL  libraries  may  currently  collect  measures  that  can  provide  some  indication  of  success 
of  a particular  program  or  service  provided  to  customers,  such  as  user  satisfaction  surveys; 
however,  it  is  important  to  think  broadly — with  the  desired  state  in  mind — and  not  simply  use 
the  measures  on  hand  because  they  are  easy  to  collect  or  because  a lot  of  time  and  effort  has 
been  devoted  to  collecting  them. 

In  most  cases,  a single  measure  on  its  own  is  not  enough  to  indicate  whether  a research 
library  is  successful  in  a given  area.  To  accurately  assess  the  success  or  quality  of  an 
academic  library,  measurement  should  be  implemented  at  three  key  levels: 
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Outcome  Level; 

Use/Capacity  Level  (Output);  and 
Resource  level  (Input). 


Figure  5 lists  some  of  the  important  questions  that  may  be  answered  at  different  levels  of 
measurement. 


Figure  5.  Using  Measures  to  Answer  Questions  at  Different  Levels 


Outcome  Level 

Use/Capacity  Level 
(Output  Measures) 

Resource  Level 
(Input  Measures) 

What  are  the  results  of  a program 
or  process? 

How  much  is  a service,  resource, 
or  program  being  used? 

What  do  we  need  to  ensure 
success? 

How  successful  or  effective  is 
the  library? 

Who  is  using  a service,  resource, 
or  program? 

What  funding  level  is  appropriate 
or  necessary  for  a particular 
program? 

How  effective  do  customers 
perceive  your  programs  to  be? 

Why  are  people  using  a particular 
program? 

Do  we  need  more  of  a particular 
resource  in  order  to  have  a more 
effective  program? 

What  beneficial  effects  are  you 
having  on  your  customers? 

How  could  a program  be  changed 
to  better  suit  the  needs 
of  your  customers? 

The  following  example  illustrates  this  approach: 

Imagine  that  information  literacy  is  a theme  for  your  university.  How  can  you  have  a 
meaningful  impact  in  this  area?  One  way  might  be  an  effective  bibliographic 
instruction  (BI)  program  that  reaches  many  students.  Ask  the  question,  “What  would 
constitute  success  for  our  BI  program?”  Once  you  have  articulated  the  answer,  you 
can  begin  to  develop  measures  for  the  effectiveness  of  your  BI  program  (i.e.,  professor 
perception  of  student  performance  before  and  after  BI  program,  etc.). 

It  is  also  important  to  know  how  many  students  can  be  reached  with  the  program,  so  capacity 
and  use  measures  are  needed  as  well.  Because  the  program  needs  staff  and  materials  need  to 
be  created,  it  is  important  to  have  resource  measures  that  indicate  what  goes  into  the  program 
and  whether  more  or  different  resources  make  a difference  in  the  effectiveness  or  success  of 
your  program. 

Here  is  another  example  in  the  networked  context: 

Imagine  that  your  university  wants  to  attract  a “world  class  faculty”.  There  are  many 
ways  that  the  library  can  have  an  impact  in  this  area.  Capitalizing  on  resources  unique 
to  your  university  is  one  way  to  “brand”  your  library  (and  in  turn  university)  as  having 
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expertise  in  a particular  field  (or  many  fields).  By  digitizing  collections  and  making 
them  widely  available,  the  library  can  attract  scholars  all  over  the  world  to  these 
materials.  As  a result,  scholars  will  begin  to  associate  these  unique  resources  with 
your  university  and  may  even  be  attracted  to  your  faculty.  There  are  many  ways  to 
measure  whether  or  to  what  extent  there  is  a linkage.  For  instance,  you  could  survey 
new  faculty  members  and  ask  if  they  used  your  library’s  materials  before  coming  on 
board,  which  materials  they  used  and  for  what,  and  whether  this  influenced  their 
decision  to  join  the  faculty. 

In  this  example,  it  is  important  to  know  who  is  accessing  the  collection,  and  which  parts  of  the 
collection  they  are  using  most  often.  This  can  indicate  which  parts  the  digital  collection  users 
are  most  interested  in  and  what  collections  may  need  to  be  more  fully  developed. 

Additionally,  it  would  be  helpful  to  know  what  resources  are  devoted  to  creating  this  digital 
collection,  and  if  increasing,  changing,  or  upgrading  resources  has  an  impact  on  the  desired 
outcome. 

Following  this  approach,  however,  may  lead  to  the  formulation  of  a wide  range  of 
performance  measures  and  statistics.  Selection  of  the  precise  measures  needed  to  evaluate  an 
electronic  resource  or  service  can  be  especially  difficult,  even  for  libraries  that  have 
undertaken  processes  similar  to  those  described  above.  Therefore,  it  is  important  to  have  a 
framework  to  assist  in  choosing  measures  to  gain  insights  into  the  use  and  uses,  management, 
and  reach  of  networked  services  and  resources  in  specific  areas  or  across  a number  of  areas. 


Challenges  Related  to  the  Use  of  Institutional  and 
Library  Outcomes 

ARL  libraries  need  to  develop  a process  to  identify  and  operationalize  library  outcomes 
that  contribute  to  institutional  outcomes.  The  library  must  play  a major  role  in  informing  the 
university  of  valued  institutional  outcomes  to  which  the  library  contributes.  Setting  up  such  a 
process  is  an  important  method  for  informing  key  stakeholders  in  the  university  of  both  the 
library’s  role  in  institutional  outcomes  and  insuring  that  the  institutional  outcomes  to  which 
the  library  has  (or  may  have)  links  are  appropriate. 

At  some  level,  this  process  will  be  developed  and/or  refined  collectively  over  time  by  the 
ARL  membership  and  related  academic  library  associations.  However,  at  a fundamental 
level,  the  process  and  implementation  is  necessarily  local,  and  the  need  for  better  orienting 
internal  library  operations  to  external  outcomes  may  be  at  hand  for  some  institutions. 

Because  each  university  has  different  processes  for  information  sharing,  decisionmaking, 
and  mission  fulfillment,  it  is  important  that  each  university  library  identify,  understand,  and 
master  the  established  local  process.  It  is  a given  that  libraries  must  work  within  their 
particular  organizational  framework.  Therefore,  to  maximize  contributions  to  university 
outcomes,  the  library  must  orient  itself  within  its  local  framework,  and  apply  that  situation  to 
its  fullest  advantage. 
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An  important  factor  that  contributes  to  an  effective  understanding  of  the  local  situation  is 
sensitivity  to  the  differing  points  of  view  of  various  stakeholder  groups.  What  might 
constitute  institutional  outcomes,  or  appropriate  institutional  outcomes,  will  vary  from  one 
group  to  another  (deans,  faculty,  trustees,  students,  etc.).  It  may  be  that  the  outcomes  from 
the  library  (as  agreed-upon  by  library  staff  and  administration)  may  or  may  not  be  those  seen 
by  university  administration  as  important  or  appropriate.  Developing  a process  to  address 
these  potentially  conflicting  stakeholder  concerns  at  the  local  level  is  particularly  vital. 

When  this  component  of  the  study  began,  we  developed  and  refined  a number  of  questions 
for  use  in  structured  interviews  and  discussion  forums.  They  were  designed  to  help  the  study 
team  better  understand  the  general  issues  and  particular  circumstances  affecting  a variety  of 
institutions.  However,  asking  and  answering  many  of  the  questions  will  also  help  libraries 
when  conducting  an  environmental  scan  or  engaging  in  a strategic  planning  process.  We 
include  those  relevant  questions  below,  somewhat  modified: 

• Is  there  a culture  of  assessment  at  your  university?  At  your  library? 

• How  does  your  university  articulate  its  core  values? 

o Are  these  values  clear?  Defined?  Measurable? 
o Are  these  values  clearly  articulated  in  the  context  of  the  library? 

• Does  your  university  measure  itself — its  outcomes — in  terms  of  its  core  values? 

o How? 

o What  measures/statistics/indicators  does  your  university  routinely  collect? 
o How  does  the  analysis  of  this  data  reflect  the  values  of  the  institution? 
o How  does  your  university  administration  use  its  outcomes  data  and  analysis  to 
change  and  improve  its  operations? 

o How  does  your  university  administration  use  its  outcomes  data  and  analysis  to 
articulate  need  for  improvements  or  changes  in  the  operations  of  the  library? 
o How  does  your  library  use  university  outcomes  data  and  analysis  to  improve 
the  operations  of  the  library? 

• Has  the  culture  of  assessment  remained  constant  at  your  university  (and  at  your 
library),  or  has  it  changed  relatively  recently? 

o If  it  has  changed,  what  were  the  causes  of  the  changes?  Does  it  change  often? 
o If  it  has  remained  constant,  does  this  reflect  rigidity  in  the  thinking  of  the 
administration  as  a whole?  Does  it  reflect  helpful  stability? 

• What  does  your  university  expect  from  the  library  in  terms  of  contributing  to 
university  outcomes? 

o Does  the  university  make  these  expectations  clear? 
o What  do  they  need  to  know  to  make  them  clearer? 

• What  does  your  university  expect  from  the  library  in  terms  of  reporting  data? 
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• How  receptive  do  you  believe  your  university  administration  is  or  would  be  to 
library  reporting  based  on  outcomes  assessment? 

• Does  your  library  currently  focus  on  campus-wide,  university-based  outcomes? 

o If  yes,  how  does  your  library  determine  which  outcomes  to  focus  on? 
o How  are  you  linking  or  matching  the  data  you  collect  with  those  outcomes? 
o How  do  you  identify  those  relationships? 

• How  do  you  see  the  way  you  assess  your  library’s  performance  changing  in  the  next 
few  years? 

o Why? 

o How  should  it  change? 

• Assuming  your  library  does  not  already  do  so,  if  your  library  were  to  measure  and 
report  its  data  in  terms  of  university  outcomes,  would  that  affect  the  way  the  library 
is  viewed  and  funded  by  your  university's  administration? 

o If  yes,  how  would  it  change? 
o Why? 

o How  difficult  would  it  be  to  effect  that  change? 

• What  are  the  key  activities  that  your  library  does  to  support  the  research,  education, 
and  service  goals  of  your  university? 

o What  kind  of  formal  or  informal  data  does  the  library  collect  that  lets  you 
know  you  are  supporting  these  goals? 

o How  does  this  play  a part  in  determining  the  types  of  resources  and  services 
you  offer? 

o How  could  this  play  a greater  role? 

• Does  your  library  collect  data  on  its  outcomes — and/or  on  university  outcomes — that 
occur  outside  the  library’s  domain? 

o If  yes,  how? 

o If  no,  is  it  clear  how  that  could  be  done  at  your  institution? 
o What  obstacles  do  you  know  of — or  perceive  or  expect — regarding  collecting 
university  outcome  data  across  the  campus? 

• Looking  at  our  framework,  does  it  meet  with  your  understanding  of  the  way  in 
which  your  library  interacts  with  the  institution  in  which  it  sits? 

o If  yes,  are  there  any  critical  details  or  factors  that  are  missing? 
o If  no,  what  is  different  at  your  institution? 

We  intend  and  hope  that  these  questions  will  help  to  stimulate  the  process  of  outcome- 
oriented  organizational  change  at  all  academic  research  institutions. 
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The  Need  for  New  Perspectives  and  Additional  Research 

Although  the  thrust  of  the  E-Metrics  project  was  on  developing  statistics  and  measures  for 
academic  and  research  libraries  in  the  networked  environment,  the  study  team  also 
investigated  institutional  outcomes  in  higher  education  and  the  role  of  the  library  in 
facilitating  the  accomplishment  of  such  outcomes.  The  study  team’s  work  in  this  area  over 
the  past  year  underscores  the  need  for  new  perspectives  and  assumptions  regarding  outcome- 
based  assessment  and  the  need  to  continue  research  in  this  area.  The  outcomes  framework 
offered  here  is  a first  step. 

To  some  degree,  the  importance  placed  on  institutional  and  library  outcomes  vary  by 
institution  and  by  its  stakeholders.  Some  institutions  of  higher  education  believe  an 
outcomes-oriented  view  toward  planning  and  evaluation  is  essential — others  do  not.  There 
are  widespread  views  about  the  usefulness,  application,  and  need  for  such  an  approach.  Thus, 
there  is  a great  need  for  empirical  studies  that  can  pursue  a number  of  the  topics  and  issues 
outlined  in  this  paper.  Such  studies  could  demonstrate  to  what  extent  and  in  what  manner 
higher  education  institutions  (including  libraries)  can  be  more  effective  and  have  a greater 
impact  (however  defined)  because  of  an  outcomes-based  approach. 

Finally,  it  is  important  to  note  that  an  outcomes-based  perspective  might  best  be  seen  as 
but  one  of  a number  of  types  of  approaches  to  support  planning  and  evaluation  in  higher 
education.  Traditional  input-output  models  of  statistics  and  measures,  goal-based  assessment, 
service  quality  approaches,  expert-based  standards,  and  others  can  be  used  together  as  a 
means  for  improving  the  overall  performance  of  institutions  of  higher  education  and  their 
various  units.  Indeed,  additional  research  that  explores  how  best  to  integrate  these  and  related 
approaches  might  be  most  fruitful  to  pursue  in  the  future. 
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Appendices 


Appendix  A:  Additional  Resources 


The  following  table  contains  a list  of  other  outcomes-related  resources  for  those  interested  in 
reading  more  about  library  and  institutional  outcomes  and  assessment: 


Program/Resource 

Author/ 

Organization 

Availability 

An  Action  Plan  for  Outcomes 
Assessment  in  Your  Library 
(2002) 

Peter  Hemon  and 
Robert  E.  Dugan 

Chicago  and  London:  American  Library  Association. 

Assessment  and  Testing: 
Measuring  Up  to  Expectations. 

Christine  Cress 

http://www.ed.eov/databases/ERIC  Dieests/ed39 1 559.html 

Assessment  in  Higher  Education 

Raymond  Walters 
College 

http://www.  rwc.uc.edu/Dhillips/Assessment/HisherEd.html 

Basic  Guide  to  Outcomes-Based 
Evaluation  for  Nonprofit 
Organizations  with  Very 
Limited  Resources 

Carter  McNamara 

http://www.mapnp.ore/librarv/evaluatn/outcomes.htm 

Internet  Resources  for  Higher 
Education  Outcomes 
Assessment 

NC  State  University: 
University  Planning 
and  Analysis 

http://www2.acs.ncsu.edu/UPA/assmt/resource.htm 

New  Measures  Initiative 

Association  of 
Research  Libraries 

http://www.arl.ore/stats/newmeas/newmeas.html 

Perspectives  on  Outcomes  Based 
Evaluation  For  Libraries  and 
Museums 

Institute  of  Museum 
and  Library  Services 
(IMLS) 

http://www.imls.eov/pubs/pdf/pubobe.pdf 

Measuring  What  Matters:  A 
Library/LRC  Outcomes 
Assessment  Manual  (2000) 

Bonnie  Gratch 
Lindauer 

[Fairfield,  CA]:  Learning  Resources  Association  of 
California  Community  Colleges. 

Task  Force  on  Academic 
Library  Outcomes  Assessment 
Report  (1998) 

Association  of 
College  and  Research 
Libraries 

http://www.ala.ore/acrl/outcome.html 

Workbook:  Outcome 
Measurement  of  Library 
Programs 

Division  of  Library 
and  Information 
Services,  Florida 
Department  of  State 

http://dlis.dos.state.fl.us/bld/Research  Office/OutcomeEval 
Wkbk.doc 

459 


January  2002 


28 


ARL  E-Metrics  Project 


Toward  a Framework  of  Outcomes 


Appendix  B:  Accreditation  Review  Methodology  and  Documents 


The  review  of  accreditation  policies  conducted  for  this  study  involved  a content  analysis 
of  the  standards  and  pertinent  supplemental  documentation  of  the  six  regional  accrediting 
commissions  of  higher  education  that  accredit  senior  colleges  and  universities.  It  addressed 
the  following  points: 

• Identification  of  overall  trends  in  accreditation  affecting  libraries  and  information 
resources; 

• A comparison,  with  examples  of  supporting  text,  of  how  outcomes  are  represented, 
contextualized  and  operationalized  with  a particular,  but  not  exclusive,  focus  on 
libraries; 

• A description  of  how  the  reviewed  standards  and  supplemental  documentation 
reference  electronic  and  networked  services,  particularly  the  need  for  university 
libraries  to  demonstrate  the  outcomes  of  these  services  and  the  need  to  show  the 
connection  between  use  of  electronic  and  networked  services  and  fulfillment  of 
their  missions;  and 

• Observations  and  suggested  recommendations  for  university  libraries  based  on  this 
review  and  analysis. 

Typically  the  standards  are  part  of  a “Handbook  of  Accreditation”  that  contains  a 
description  of  the  process,  the  eligibility  requirements,  relevant  policies  that  institutions  must 
address  in  their  self-study  reports,  and  other  documentation  developed  to  assist  institutions 
with  preparing  their  self-studies  and  conducting  evaluation  and  assessment  studies.  Only  the 
Northwest  Association  of  Schools  and  Colleges  (Northwest)  integrates  its  policies  within  the 
standards  by  locating  them  at  the  end  of  the  standard  to  which  they  relate.  Some  of  the 
documents  listed  below  “required”  supporting  documentation  and/or  “suggested”  supporting 
documentation  after  each  standard  (e.g.,  Western  Association  of  Schools  and  Colleges 
(Western),  Middle  States  Association  of  Colleges  and  Schools  (Middles  States),  and 
Northwest).  All  of  them  provide  guidance  either  within  the  handbook  or  as  separate  policies 
for  improving  assessment  and  evaluation  of  institutional  effectiveness  and  student  learning. 

The  review  included  both  current  and  proposed  draft  versions  of  the  standards.  Since  the 
time  the  review  was  completed,  the  Southern  Association  of  Colleges  and  Schools  (Southern) 
was  expected  to  have  a near  final  draft  on  its  website  in  August  2001  with  final  approval  by 
December  2001.  As  of  January  7,  2001,  according  to  the  official  Southern  website,  its 
Commission  on  Colleges  adopted  the  draft  in  June  2001,  and  final  adoption  was  scheduled  for 
December  1 1,  2001;  however,  no  updates  on  the  proposed  adoption  have  yet  been  posted. 
Meanwhile,  Northwest  and  the  New  England  Association  of  Schools  and  Colleges  (New 
England)  Commissions  are  not  currently  working  on  revised  standards.  Middle  States  expects 
to  have  a final  and  approved  copy  by  early  2002. 

The  findings  discussed  in  this  paper  reflect  the  text  of  the  standards  and  pertinent 
supporting  documentation  such  as  policies  and  related  documents.  The  process  of 
accreditation  usually  involves  several  stages  for  institutions  seeking  accreditation  and  re- 
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accreditation.  Part  of  this  process  involves  institutions  documenting  their  compliance  with 
basic  eligibility  requirements.  All  of  the  six  commissions’  documents  reviewed,  except  that 
of  New  England,  contain  eligibility  requirements.  Eligibility  requirements  relating  to  libraries 
and  networked  information  sources  are  typically  vague  and  input-specific.  Therefore,  it  is 
particularly  noteworthy  that  Western’s  requirement  specifies  having  programs  for  student 
training  and  instruction  in  information  literacy  and  that  institutions  “must  also  be  able  to 
demonstrate  that  library  use  is  a fundamental  part  of  all  curricula”  (Western,  How  to  Become 
Accredited , p.  4). 

The  following  documents  were  analyzed  for  this  paper  and  are  listed  here  by  issuing 
institution: 

Middle  States  Association  of  Colleges  and  Schools 

(http://www.msache.org) 

1.  Characteristics  of  Excellence  in  Higher  Education : Standards  for  Excellence  in 

Higher  Education  (1994).  Commission  on  Higher  Education. 

2.  Characteristics  of  Excellence  in  Higher  Education.  (February  22,  2001  Draft). 

3.  Designs  for  Excellence:  Handbook  for  Institutional  Self-Study , 7th  ed.  (2000). 

4.  Framework  for  Outcomes  Assessment.  (1996).  Commission  on  Higher  Education. 

New  England  Association  of  Schools  and  Colleges 

(http://www.neasc.org) 

5.  “Pilot  Institutional  Assessment  Portfolios.”  (2001).  Commission  on  Institutions  of 

Higher  Education. 

6.  Standards  for  Accreditation.  (1992).  Commission  on  Institutions  of  Higher 

Education. 

North  Central  Association  of  Colleges  and  Schools 

(http://www.ncahigherleamingcommission.org) 

7.  Addendum  to  the  Handbook  of  Accreditation,  2nd  ed.  (March  2001).  The  Higher 

Learning  Commission,  www.ncahigherleamingcommission.org.  This  includes 
the  text  of  the  updated  March  2001  “Criteria  for  Accreditation”,  current 
policies  and  a document  called  “Assessment  of  Student  Academic 
Achievement:  Levels  of  Implementation”  (updated  March  2001). 

8.  ’’Statement  of  Commitment  by  the  Regional  Accrediting  Associations  for  the 

Evaluation  of  Electronically  Offered  Degree  and  Certificate  Programs”  and 
“Best  Practices  for  Electronically  Offered  Degree  and  Certificate  Programs” 
(Sept.  2000).  The  “Statement  of  Commitment...”  was  developed  by  the  eight 
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regional  accrediting  commissions  and  the  “Best  Practices...”  are  based  on  a 
document  by  the  same  title  initially  drafted  by  the  Western  Cooperative  for 
Educational  Telecommunications.  Both  are  located  in  the  North  Central 
Association’s  Handbook  of  Accreditation,  pp.  44-54. 

Northwest  Association  of  Schools  and  Colleges 

(http://www.cocnasc.org) 

9.  Standards  of  Accreditation.  (1999).  Commission  on  Colleges. 

Southern  Association  of  Colleges  and  Schools 

(http://www.sacscoc.org/) 

10.  Criteria  for  Accreditation.  (1998).  Commission  on  Colleges. 

1 1.  “Distance  Education:  Definition  and  Principles — A Policy  Statement.  (Adopted 

June  1997;  Updated  May  2000).  Commission  on  Colleges. 

12.  Principles  of  Accreditation.  (March  2001  Draft).  Commission  on  Colleges. 

Western  Association  of  Schools  and  Colleges 

(http://www.wascweb.org/) 

13.  Handbook  of  Accreditation.  (January  2001).  Accrediting  Commission  for  Senior 

Colleges  and  Universities.  This  includes  the  standards,  policies  and  procedures 
required  by  WASC. 

14.  How  to  Become  Accredited:  Procedures  Manual  on  Eligibility,  Candidacy,  and 

Initial  Accreditation,  (n.d.).  Accrediting  Commission  for  Senior  Colleges  and 
Universities. 
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Appendix  C:  Survey  Instrument 


Developing  Statistics  and  Performance  Measures  to  Describe  Electronic 
Information  Services  and  Resources  for  ARL  Libraries 
http://www.arl.org/stats/newmeas/emetrics/ 

Survey  of  Library  and  University  Outcomes 

This  survey  is  being  conducted  by  the  Information  Use  Management  and  Policy  Institute 
(Information  Institute)  at  the  School  of  Information  Studies,  Florida  State  University,  as  part 
of  the  E-Metrics  Project  funded  by  24  ARL  member  libraries.  ARL  is  distributing  this  as  a 
SPEC  Survey  but  will  provide  responses  to  the  Information  Institute  project  team  for  analysis 
within  its  project,  and  ARL  will  not  provide  a separate  SPEC  Kit.  Findings  from  this  survey 
will  inform  the  project  team’s  ongoing  study  of  outcomes  in  the  context  of  libraries  in 
research  universities.  More  information  about  this  project  can  be  found  on  the  Information 
Institute  website  <http://www.ii.fsu.edu/Proiects/ARL/>  or  in  the  ARL  New  Measures 
Initiative  website  <http://www.arl.org/stats/newmeas/emetrics/>. 


The  purpose  of  this  survey  is  to  gain  insights  into  the  various  factors  affecting  outcomes 
assessment  in  ARL  member  libraries  within  universities.  {Note:  If  your  library  is  not  part  of 
a university,  we  ask  that  you  do  not  complete  this  survey  as  it  does  not  apply  to  your 
organization.)  By  “outcomes”  we  mean  those  specific  outcomes  of  interest  to  libraries  and 
their  universities  that  flow  from  the  mission-  and  goal-oriented  work  of  those  organizations, 
outcomes  which  give  meaningful  indications  of  whether  the  mission  and  goals  are  being 
achieved.  The  project  team  is  interested  in  better  understanding  how  universities  and  their 
libraries:  (a)  set  their  strategic  directions  with  both  institutional  and  library  outcomes  of 
interest;  (b)  articulate  those  outcomes  and  the  results  desired;  (c)  assess  whether  the  desired 
outcomes  have  been  achieved;  and  (d)  use  outcomes-related  concerns  and  data  in  decision 
making. 


Your  participation  will  involve  completing  this  survey  and,  if  the  information  is  readily 
available  to  you,  it  should  take  approximately  30-45  minutes  to  complete  the  open-ended 
questions  (identifying  and  sending  the  requested  documents  may  take  longer).  Your 
participation  in  this  survey  is  voluntary  and  you  may  withdraw  at  any  time.  There  is  no 
penalty  if  you  choose  not  to  answer  any  or  all  of  the  questions.  Your  response  to  this  survey 
will  remain  confidential  to  the  extent  allowable  by  law.  The  results  of  the  study  may  be 
published,  but  any  individual  responses  used  will  not  be  accompanied  by  identifying 
information.  Documents  collected,  however,  may  be  quoted  and/or  cited  using  the  name  of 
the  originating  university.  The  study  team  may  contact  willing  survey  participants  for  follow- 
up questions  to  gain  more  in-depth  insights  into  their  responses. 

The  study  team  asks  that  you  complete  this  survey  as  soon  as  possible,  and  prior  to  July  6, 
2001.  Should  you  have  any  questions  concerning  the  research  project,  please  contact  Bruce 
T.  Fraser  by  telephone  at  (850)  645-3328,  or  by  e-mail  at  bfraser@lis.fsu.edu. 
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Name  of  Institution: 


Name  of  Respondent: 


Title  of  Respondent: 


Email: 


Phone: 


Open-Ended  Questions 


Please  answer  the  following  questions,  in  as  much  detail  as  possible,  using  the  text  boxes 
provided.  Thank  you  for  your  time  and  effort. 


1.  Institutional  Outcomes: 

a.  What  requirements  do  you  receive  from  your  university  administration, 
accreditation  board,  faculty,  legislature,  and  board  of  regents,  etc.,  related  to 
the  key  work  of  your  library  and  how  it  should  be  assessed?  Please  specify  in 
as  much  detail  as  possible. 
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b.  What  other  directions,  signals  and  cues  are  you  receiving  from  (1)  those  listed 
above  and  (2)  other  stakeholder  groups  that  influence  your  view  of  the  key 
work  of  your  library  and  how  it  should  be  assessed?  For  example,  apart  from 
formal  requirements  has  your  university  administration  communicated  any 
themes  (such  as  economic  development  in  your  state)  that  every  campus  unit  is 
expected  to  support?  Please  specify  in  as  much  detail  as  possible. 


2.  Library  Operations/Processes: 

How  do  the  factors  you  listed  in  question  la.  and  lb.  affect  the  planning, 
budgeting,  acquisitions,  and  assessment  of  your  library?  Please  specify  in  as 
much  detail  as  possible. 


3.  Outcomes  Assessment: 

a.  How  does  the  key  work  of  your  library  impact  the  key  work  of  the  university? 
In  other  words,  what  does  your  library  do  that  can  be,  or  is,  linked  to  the 
research,  education,  and  service  outcomes  of  your  university?  Please  specify 
in  as  much  detail  as  possible. 
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b.  How  do  you  know  what  impact  your  library  has  on  your  university  and  its 
mission?  (Customer  surveys,  performance  measurement,  informal 
communications,  intuition,  etc.)  Please  specify  in  as  much  detail  as  possible. 


c.  How  do  you  articulate  this  to  the  Provost  or  other  university  administrators  and 
deans?  Please  specify  in  as  much  detail  as  possible. 
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Documents 


In  addition  to  the  survey,  please  share  any  documents  that  will  help  us  to  understand  how 
expectations  are  made  clear  to  libraries  by  their  stakeholders,  how  libraries  are  reacting, 
and  how  the  interplay  affects  decisions — especially  those  related  to  budgeting — at  the 
library  and  university  levels.  Two  such  documents  are  the  university  and  library  strategic 
plans.  These  important  documents  can  contain  information  about  what  the  library  and 
university  value  and  how  they  are  situating  themselves  in  the  rapidly  changing  higher 
education  landscape. 

Other  documents  of  interest  include,  but  are  not  limited  to,  memos  from  the  Provost  (or 
other  key  university  administrators)  regarding  the  way  budget  decisions  are  made,  library 
documents  pertaining  to  outcomes  assessment/performance  measurement,  and  academic 
program  review  documents  that  mention  library  holdings/resources.  We  would  like  to 
thank  those  libraries  that  have  already  provided  us  with  such  documents  in  this  project  and 
encourage  them  to  submit  any  additional  items  that  may  be  useful. 

To  submit  documents  that  can  be  found  on  the  Web,  please  enter  the  URLs  of  the 
documents  in  the  boxes  provided.  If  certain  documents  are  not  available  on  the  Web,  but 
you  will  be  submitting  them,  please  check  the  space  labeled  “We  will  send  this  document 
via  email,  mail,  or  fax.” 

■ Please  email  documents  to: 

Bruce  T.  Fraser 
bfraser@lis.fsu.edu. 


■ Please  fax  documents  to: 

(850)  644-9763 
Attn:  Bruce  T.  Fraser 

■ Please  mail  documents  to: 

Bruce  T.  Fraser 
Information  Institute 
226  Louis  Shores  Building 
Florida  State  University 
Tallahassee,  FL  32306-2100 


1 . University  strategic  plan  (or  similar  documents  that  pertain  to  vision,  mission,  goals, 
and/or  indicators/measures) 
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Please  identify  URLs 


We  will  send  this  document  via  email,  mail,  or  fax. 

2.  Library  strategic  plan  (or  similar  documents  that  pertain  to  vision,  mission,  goals, 
and/or  indicators/measures) 

Please  identify  URLs 


We  will  send  this  document  via  email,  mail,  or  fax. 


3.  Other  documents  from  the  Provost  or  others  regarding  strategic  planning 
Please  identify  URLs 


We  will  send  this  document  via  email,  mail,  or  fax. 
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4.  Other  documents  from  the  Provosts  or  others  regarding  budgeting 
Please  identify  URLs 


We  will  send  this  document  via  email,  mail,  or  fax. 


5.  Other  university  and/or  library  documents  regarding  outcomes,  performance  measurement, 
etc. 


Please  identify  URLs 


We  will  send  this  document  via  email,  mail,  or  fax. 


Thank  you  for  your  time  and  effort  and  interest  in  this  study.  Please  check  the  following  box 
if  you  are  willing  to  be  contacted  by  a member  of  the  project  team  for  follow-up  questions: 

□ 1 am  willing  to  be  contacted  by  a member  of  the 

project  team  for  follow-up  questions. 
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Appendix  D:  Content  Analysis  of  the  Survey  by  Question 


In  the  question-by-question  discussion  of  the  survey  responses  that  follow,  we  summarize 
in  each  case  the  general  sense  of  the  responses,  highlighting  selected  examples  that  are  either 
representative  or  otherwise  noteworthy.  Not  all  of  the  documents  supplied  spoke  directly  to 
each  specific  question,  but  often  they  did  illustrate  many  of  the  points  raised.  When  helpful, 
quotations  without  attribution  (for  the  sake  of  confidentiality)  from  open-ended  responses  or 
documents  are  used  below. 

la.  What  requirements  do  you  receive  from  your  university  administration,  accreditation 
board \ faculty , legislature,  and  board  of  regents,  etc.,  related  to  the  key  work  of your 
library  and  how  it  should  be  assessed? 

Most  of  the  ARL  libraries  responded  that  there  are  few  assessment  requirements  related  to 
the  library,  if  any,  set  forth  by  the  university  administration  or  others.  Though  requirements 
may  not  be  set  forth  as  a general  rule,  a variety  of  circumstances  exist  across  the  different 
campuses,  and  at  a given  campus  at  different  times.  Some  respondents  noted  a freedom  to 
assess  the  library  with  great  discretion,  and  many  indicated  they  provide  annual 
accomplishment  or  budget  reports  to  university  administrators  that  list  achievements  for  a 
given  year.  One  stated  that  while  having  “no  formal/legislated  requirements  regarding  the 
key  work  of  the  library,  the  four  year  budget/strategic  plan  for  the  library,  along  with  its 
consequent  objectives  and  activities,  must  be  linked  ...  to  the  university’s  key  strategic 
initiatives,  [with]  performance  indicators”  (emphasis  added).  Others  mentioned  performing 
self-studies  and/or  answering  to  review  boards. 

lb.  What  other  directions,  signals  and  cues  are  you  receiving  from  (1)  those  listed  above 
and  (2)  other  stakeholder  groups  that  influence  your  view  of  the  key  work  of  your  library 
and  how  it  should  be  assessed? 

Two  of  the  universities  give  specific  directions,  signal  or  cues.  Others  admit  that  there  is  a 
certain  freedom  because  of  their  “modest  formal  structure,”  the  fact  that  they  are  private 
universities,  or  because  the  “university  lacks  a strong  statement  of  strategic  objectives.”  One 
interesting  observation  was  that  one  of  the  libraries  looks  at  its  interaction  with  other  ARL 
members  in  order  to  determine  what  directions  should  be  taken,  as  well  as  looking  towards 
organizations  like  the  Digital  Library  Federation.  Feedback  from  students  and  faculty  was 
also  cited  as  a source  of  important  signals.  Three  responded  by  pointing  out  they  are  land 
grant  institutions  and  that  this  plays  a significant  role  in  determining  what  directions,  signals 
and  or  cues  are  sent  to  the  library.  Two  respondents  mentioned  interactions  with  the 
university  president  being  important  in  this  context. 

2.  How  do  the  factors  you  listed  in  question  la  and  lb  affect  the  planning,  budgeting, 
acquisitions,  and  assessment  of  your  library? 

All  of  the  respondents  stated  that  in  some  way  the  requirements,  directions,  signals  and 
the  like  have  a direct  effect  on  the  planning,  budgeting,  acquisitions,  and/or  assessment  of 
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their  libraries,  though  the  relationship  is  not  always  clear  or  precise.  One  stated  that  the 
impact  was  only  moderate.  Another  offered  that:  “They've  made  the  library  a more  effective 
organization  because  it  now  must  articulate  clearly  its  goals  and  priorities  and  both  assess 
itself  and  be  assessed  on  how  well  those  are  met.  In  a new  undertaking,  an  outgrowth  of  a 
campus  committee,  our  AUL  for  Collections  will  meet  with  people  designated  by  each 
collegiate  dean  to  discuss,  in  detail,  the  College's  priorities.” 

3 a.  How  does  the  work  of  your  library  impact  the  key  work  of  the  university?  In  other 
words,  what  does  your  library  do  that  can  or  is,  linked  to  the  research,  education,  and 
service  outcomes  of  your  university?  Please  specify  in  as  much  detail  as  possible. 

More  than  half  of  the  respondents  indicated  that  this  is  either  “unmeasured”  or  “unknown” 
because  this  process  is  “really  hard  to  do  in  any  quantitative  way,”  as  one  pointed  out. 

Another  was  “not  convinced  that  this  is  really  possible.”  However,  many  did  offer  a number 
of  impacts,  some  matter-of-factly  and  others  with  explanations.  In  either  case  an  academic 
library’s  role  for  its  university’s  key  work  may  vary  depending  on  the  institutional  goals.  The 
roles  of  university  and  research  libraries  are  “geared  to  supporting  the  teaching,  research,  and 
service  needs”  of  survey  participants’  universities.  To  illustrate,  one  library  and  its  institution 
tries  to  anticipate  the  needs  of  faculty  and  students  by  providing  “the  most  efficient  way  to 
access  that  information,  including  delivering  materials  to  offices  for  faculty  and  to  other 
libraries  on  grounds  for  students;  providing  electronic  databases,  and  easy  access  to  them,  on 
or  off-grounds;  providing  research  and  reference  assistance  through  a variety  of  means; 
providing  collections  relevant  to  researcher's  needs  in  several  media;  providing  a variety  of 
digital  services;  providing  User  Education  courses  and  Short  Courses  in  library  and 
technology  skills;  providing  free  basic  services  to  users  from  the  local  community.”  In  other 
words,  the  role  of  the  library  is  “to  acquire  relevant  collections,  to  provide  access  to  them,  to 
provide  the  tools  of  technology  to  access  them,  and  to  provide  instruction  in  their  use.” 

The  key  work  that  the  library  does  may  influence  the  university’s  institutional  outcomes 
of  interest  “by  providing  portals  to  information  which  results  in  the  production  of  original 
research.”  One  respondent  noted  that  libraries  “provide  information  resources  for  essential 
academic  and  research  programs;  instruct  students  in  the  identification,  evaluation,  and  use  of 
such  resources;  collaborate  with  faculty  and  serve  as  a research  test  bed  for  innovative 
information  technology  or  information  resource  development  projects;  contribute  staff  time 
and  expertise  to  participating  in  the  university  community;  provide  a valuable  service  on 
behalf  of  the  university  to  the  external  community.” 

As  some  libraries  reported,  they  have  started  several  initiatives  involving  students  and  key 
faculty  in  developing  or  acquiring  collections,  in  creating  digital  collections,  and  in  designing 
services,  thereby  insuring  that  they  will  provide  some  definite  support  for  at  least  some 
stakeholders’  wants  and  needs.  For  example,  the  desired  “emphasis  can  be  on  special 
collections,  such  as  international  collections,  supporting  international  studies  and  many  global 
initiatives.” 
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3b.  How  do  you  know  what  impact  your  library  has  on  your  university  and  its  mission? 

(Customer  surveys,  performance  measurement,  informal  communications,  intuition,  etc.) 

Please  specify  in  as  much  detail  as  possible. 

There  are  various  ways  reported  for  what  impact  libraries  have  on  their  university 
institutions  and  their  mission.  Informal  communications  were  reported  to  be  one  of  the  ways 
to  become  knowledgeable  about  what  impact  libraries  have  on  their  university  missions.  User 
surveys,  both  “broad-based  and  targeted”,  are  widely  used  on  campuses  to  know  what  impact 
libraries  have  impact  on  university  communities,  employing  LIBQUAL+,  ServQual  and 
student  surveys.  In  addition  to  informal  communication,  intuition,  and  feedback  from  faculty 
and  administrators,  universities  use  statistics  about  numbers  of  students  reached  through 
bibliographic  instruction  classes  or  information  literacy  programs,  statistics  about  acquisitions 
growth  (monographs,  serials,  electronic  resources)  by  subject  area,  document  numbers  and 
subject  breadth  of  collaborative  projects  with  faculty  and  database  and  web  use.  Additionally, 
as  one  noted,  the  following  techniques  are  used  to  find  out  what  impact  libraries  have  on 
campuses:  Focus  groups,  usability  testing,  transaction  logs,  email  monitoring  (though  this 
was  not  explained),  grant  and  funding  procurement,  and  citation  studies. 

3c.  How  do  you  articulate  this  to  the  provost  or  other  university  administrators  and 

deans?  Please  specify  in  as  much  detail  possible. 

There  are  several  ways  of  articulating  the  impact  of  the  library  to  the  provost  or  other 
senior  university  administrators.  Respondents  reported  a number  of  activities  that  libraries  do 
to  articulate  their  needs  and  impacts  to  various  university  administrators,  compiled  as  follows: 
Announcements  and  articles  in  university,  annual  reports,  annual  briefings,  annual  state  of  the 
library  messages,  budget  request  statements,  committee  discussions,  during  the  deans’ 
retreats,  informal  notes  about  specific  events,  newsletters,  news  releases,  newspapers, 
personal  conversations,  presentations  to  various  groups  and  individuals,  including  a “Board  of 
Deans,”  statistical  reports  sent  to  university  administrators,  strategic  planning  process,  survey 
reports,  targeted  communications  (documents,  letters,  emails,  phone  calls,  etc.),  and  the  use 
of  information  in  strategic  documents. 

These  activities  of  course  vary  from  one  institution  to  another,  and  may  in  some  cases  be 
carried  out  either  formally  or  informally.  To  identify  the  needs  of  library  users,  for  example, 
one  of  the  respondent  institutions  “uses  results  of  surveys  of  incoming  freshmen  to 
demonstrate  need  for  [a]  cumulative  instructional  program.”  Almost  all  of  respondents  shared 
the  importance  of  keeping  the  provost  or  other  administrators  and  deans  updated  as  well  as 
informing  “the  chancellor  and  the  executive  Vice-chancellor,  in  University  Council,  and  in 
our  various  advisory  groups”  regularly.  As  one  respondent  said,  indicating  sensitivity  to 
possible  indirect  lines  of  communication,  a library’s  “selected  library  advisory  committees 
also  report  back  to  their  deans  (and  the  most  effective  communication  is  when  deans  hear 
these  reports  from  their  faculty  rather  than  from  the  library).” 
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I.  Introduction  and  Methodology 

This  report  reflects  the  findings  of  content  analyses  of  the  standards  and  pertinent  supplemental 
documentation  of  the  six  regional  accrediting  commissions  of  higher  education  that  accredit  senior 
colleges  and  universities.  It  addresses  the  following  points: 

• identification  of  overall  trends  in  accreditation  that  affect  libraries  and  information  resources; 

• a comparison,  with  examples  of  supporting  text,  of  how  outcomes  are  represented,  contextualized, 
and  operationalized  with  a particular,  but  not  exclusive,  focus  on  libraries; 

• a description  of  how  the  standards  and  supplemental  documentation  reference  electronic  and 
networked  services,  particularly  the  need  for  university  libraries  to  demonstrate  the  outcomes  of 
these  services  and  the  need  to  show  the  connection  between  use  of  electronic  and  networked  services 
and  fulfillment  of  their  missions; 

• observations  and  suggested  recommendations  for  university  libraries  based  on  this  review  and 
analysis. 

Typically,  the  standards  are  part  of  a “Handbook  of  Accreditation”  that  contains  a description  of  the 
process,  the  eligibility  requirements,  relevant  policies  that  institutions  must  address  in  their  self-study 
reports,  and  other  documentation  developed  to  assist  institutions  with  preparing  their  self-studies  and 
conducting  evaluation  and  assessment  studies.  Only  the  Northwest  Association  integrates  its  policies 
within  the  standards  by  locating  them  at  the  end  of  the  standard  to  which  they  relate.  Some  of  the 
documents  list  “required”  and/or  “suggested”  supporting  documentation  after  each  standard  (WASC, 
Middle  States,  Northwest).  All  of  them  provide  guidance  for  improving  assessment  and  evaluation  of 
institutional  effectiveness  and  student  learning  either  within  the  handbook  or  as  separate  policies. 


The  following  documents  were  used  in  the  content  analysis: 
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1 . Handbook  of  Accreditation  (January  200 1 ).  Accrediting  Commission  for  Senior  Colleges  and 

Universities.  Western  Association  of  Schools  and  Colleges  (WASC). 

<http://www.wascweb.org> 

This  includes  the  standards,  policies,  and  procedures  required  by  WASC. 

2.  Characteristics  of  Excellence  in  Higher  Education : Standards  for  Excellence  in  Higher  Education 

(1994).  Commission  on  Higher  Education.  Middle  States  Association  of  Colleges  and  Schools 
(MS),  <http://www.msache.org> 

3.  Framework  for  Outcomes  Assessment  { 1996).  Commission  on  Higher  Education.  Middle  States 

Association  of  Colleges  and  Schools. 

4.  Characteristics  of  Excellence  in  Higher  Education  (February  22,  2001  Draft)  and  Designs  for 

Excellence:  Handbook  for  Institutional  Self-Study,  7th  ed.  (2000).  Middle  States  Association  of 
Colleges  and  Schools,  <http://www.msache.org> 

5.  Standards  of  Accreditation  (1999).  Commission  on  Colleges.  The  Northwest  Association  of  Schools 

and  Colleges  (NW).  <http://www.cocnasc.org> 

6.  Criteria  for  Accreditation  (1998).  Commission  on  Colleges.  Southern  Association  of  Colleges  and 

Schools  (SACS),  <http://www.sacscoc.org> 

7.  Principles  of  Accreditation  (March  2001  Draft).  Commission  on  Colleges.  Southern  Association  of 

Colleges  and  Schools,  <http://www.sacscoc.org> 

8.  “Distance  Education:  Definition  and  Principles — A Policy  Statement.  (Adopted  June  1997;  Updated 

May  2000).  Commission  on  Colleges.  Southern  Association  of  Colleges  and  Schools. 

9.  Addendum  to  the  Handbook  of  Accreditation.  2nd  ed.  (March  2001).  The  Higher  Learning 

Commission.  North  Central  Association  of  Colleges  and  Schools. 
<http://www.ncahigherleamingcommission.org> 

This  includes  the  text  of  the  updated  March  2001  “Criteria  for  Accreditation,”  current  policies, 
and  a document  called  “Assessment  of  Student  Academic  Achievement:  Levels  of 
Implementation”  (updated  March  2001). 

10.  Standards  for  Accreditation  (1992).  Commission  on  Institutions  of  Higher  Education.  New  England 

Association  of  Schools  and  Colleges  (NE).  <http://www.neasc.org> 

1 1.  “Pilot  Institutional  Assessment  Portfolios”  (2001).  Commission  on  Institutions  of  Higher  Education. 

New  England  Association  of  Schools  and  Colleges.  <http://  www.neasc.org> 

12.  “Statement  of  Commitment  by  the  Regional  Accrediting  Associations  for  the  Evaluation  of 

Electronically  Offered  Degree  and  Certificate  Programs”  and  “Best  Practices  for  Electronically 
Offered  Degree  and  Certificate  Programs”  (September  2000). 

The  “Statement  of  Commitment...”  was  developed  by  the  eight  regional  accrediting  commissions 
and  the  “Best  Practices...”  are  based  on  a document  by  the  same  title  initially  drafted  by  the 
Western  Cooperative  for  Educational  Telecommunications.  Both  are  located  in  the  North  Central 
Association’s  Handbook  of  Accreditation,  pp.  44-54. 
<http://www.ncahigherleamingcommission.org> 

Both  the  current  and  proposed  draft  versions  of  the  standards  were  reviewed  and  the  status  of  the  drafts 
and  expected  dates  for  final  copy  were  confirmed  with  the  regional  accrediting  associations.  The 
Southern  Association  expects  to  have  a near  final  draft  on  its  Web  site  in  August  2001  with  final 
approval  by  December  2001.  The  Northwest  and  New  England  commissions  are  not  currently  working 
on  revised  standards.  The  Middle  States  Association  expects  to  have  a final  and  approved  copy  by  early 
2002.  Throughout  this  report,  the  date  of  the  standards  will  only  be  referenced  in  those  cases  where 
there  is  both  a current  and  draft  version  of  the  standards. 
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This  report  primarily  uses  the  text  of  the  standards  and  pertinent  supporting  documentation  such  as 
policies  and  related  documents.  The  process  of  accreditation  usually  involves  several  stages  for 
institutions  seeking  accreditation  and  re-accreditation.  Part  of  this  process  involves  institutions 
documenting  their  compliance  with  basic  eligibility  requirements.  All  of  the  six  commissions’ 
documents  reviewed — except  the  New  England  Association — contain  eligibility  requirements.  These 
basic  requirements,  along  with  text  from  the  standards  relating  to  libraries  and  electronic  resources  and 
services,  are  included  in  Figure  2 in  the  Appendix.  Eligibility  requirements  relating  to  libraries  and 
networked  information  sources  are  typically  vague  and  input-specific.  Therefore,  it  is  particularly 
noteworthy  that  the  WASC  requirement  specifies  having  programs  for  student  training  and  instruction  in 
information  literacy  and  that  institutions  “must  also  be  able  to  demonstrate  that  library  use  is  a 
fundamental  part  of  curricula.” 

II.  Overall  Trends  Related  to  Libraries  and  Information  Resources  Based  on  Contextual  Elements 
and  the  Overall  Organization  and  Conceptualization  of  the  Standards 

1.  Mission  and  goals-driven 

All  of  the  documents  emphasize  a goals-based  assessment  model  using  mission-driven  standards  to 
define  educational  quality.  They  all  explicitly  express  the  expectation  that  each  program  or  unit 
establishes  goals  which  derive  from  and  support  the  purpose  of  the  institution;  evaluates  its  success  in 
achieving  these  goals;  and  demonstrates  the  use  of  the  evaluation  findings  in  making  improvements  and 
modifications.  Clearly,  the  university  library  is  included  in  this  expectation. 

2.  Text  is  less  prescriptive  and  specific 

Overall,  the  language  used  in  the  documents  revised  since  1998  and  in  those  currently  undergoing 
revision  is  less  prescriptive  and  less  concerned  with  as  many  specific  inputs.  SACS  has  eliminated  the 
“must”  in  its  standards.  Figure  2 in  the  Appendix  compares  the  text  that  references  libraries,  information 
resources,  and  information  technology  and  provides  examples  of  the  de-emphasis  on  the  amount  of  and 
specificity  of  the  text  relating  to  library  services  and  resources.  Three  of  the  six  commissions  (WASC, 
Middle  States,  and  SACS)  have  eliminated  a separate  standard  for  libraries  and  information  resources; 
instead  statements  referencing  them  are  embedded  in  other  sections  of  the  standards.  In  these 
documents  references  to  libraries  and  information  resources  are  less  specific  than  the  previous  editions 
of  their  standards.  Comparing  the  text  in  Figure  2 for  MS  and  SACS,  for  example,  confirms  that  there  is 
less  text  devoted  to  libraries  and  information  resources  and  services  than  in  the  2001  drafts.  It  is  also 
much  less  specific,  describing  primarily  three  expectations  in  the  2001  drafts:  having  adequate  staff; 
providing  access  to  appropriate  collections,  resources  and  services;  and  ensuring  that  students  receive 
instruction  in  the  use  of  library  resources  and  information  technology. 

Only  two  commissions  (New  England  and  Northwest)  have  maintained  a separate  standard  with  six  or 
so  separate  statements  about  collections,  services,  facilities,  staffing,  access,  and  organization  and 
operations.  Moreover,  now  all  of  the  revised  or  draft  standards  are  either  somewhat  vague  about 
collection  ownership  or  explicitly  state  that  access  to  adequate  library  collections  and  resources  is 
sufficient. 

3.  More  experimental  and  collaborative  accreditation  process 

The  overall  accreditation  process  also  shows  changes  among  the  recently  revised  and  draft  documents. 
WASC,  Middle  States,  and  North  Central  have  adopted  revised  accreditation  models  that  are  more 


* North  Central  did  not  have  a separate  standard  referencing  libraries  in  the  former  edition  of  its  standards  either. 

3 


focused  on  innovation,  experimentation,  and  collaboration.  These  values  are  reflected  in  the  following 
statement  from  the  WASC  Handbook  of  Accreditation:  “to  support  multiple  models  of  institutional 
presentation  and  demonstration  of  meeting  common  core  commitments  and  standards”  and  “focus  on 
collaboration  between  WASC  and  institutions  for  developing  new  approaches.” 

The  conceptualization  and  phrasing  of  the  revised  standards  in  the  WASC  Handbook  of  Accreditation 
are  unique  among  the  six  documents  reviewed  for  reflecting  a process  model  that  promotes  desired 
outcomes  and  emphasizes  the  dynamic  nature  of  institutional  improvement: 

Standard  1 “Defining  Institutional  Purposes  and  Ensuring  Educational  Objectives” 

Standard  2 “Achieving  Educational  Objectives  through  Core  Functions” 

Standard  3 “Developing  and  Applying  Resources  and  Organizational  Structures  to  Ensure 
Sustainability” 

Standard  4 “Creating  an  Organization  Committed  to  Learning  and  Improvement” 

North  Central’s  March  2001  Addendum  to  the  Handbook  of  Accreditation  describes  its  commitment  to 
promoting  flexibility  and  innovation,  as  well  as  providing  new  services  that  share  effective  models  of 
learning  and  provide  “new  ways  to  work  in  partnership  with  stakeholders  from  higher  education  and  the 
public  to  foster  a culture  of  assessment  for  its  members.”  It  has  also  launched  an  alternative  process  for 
re-accreditation  called  the  Academic  Quality  Improvement  Project.  The  goal  of  AQIP  is  to  “design  an 
innovative,  more  challenging  alternative  to  current  re-accreditation,  one  that  engages  institutions  by 
increasing  the  tangible  benefits  it  delivers  to  them  and  based  on  quality  improvement  principles  and 
values.”  The  criteria,  questions,  and  information  about  this  initiative  are  located  at  the  Web  site 
<http://www.AQIP.org>. 

The  eight  regional  accrediting  commissions  have  drafted  a “Statement  of  Commitment  for  the 
Evaluation  of  Electronically  Offered  Degree  and  Certificate  Programs.”  The  section  “Commitment  to 
Cooperation,  Consistency  and  Collaboration”  contains  the  following  statement:  “As  each  of  the  regional 
commissions  continues  to  accrue  skills  in  assessing  distributed  education  programming,  they  are 
pledged  to  learn  from  the  experiences  of  one  another,  particularly  when  innovative  approaches  are 
utilized.” 

4.  Emphasis  on  outcomes 

The  emphasis  on  assessing  student  learning  and  other  outcomes  is  generally  stronger  in  those  standards 
that  have  been  revised  since  1998  or  are  currently  undergoing  revision.  Moreover,  Title  IV  of  the  1998 
Higher  Education  Amendments  requires  universities  that  receive  federal  monies  to  have  an  outcomes 
assessment  plan  that  includes  “a  review  of  the  institution's  success  with  respect  to  student  achievement 
in  relation  to  mission.  Institutions  should  include  in  the  self-study  a review  of  course  completion, 
graduation  rates,  state  licensure  exam  pass  rates,  and  other  data  as  appropriate  to  the  mission  of  the 
institution  and  the  programs  it  offers.”  {Design  for  Excellence:  Handbook  of  Institutional  Self-Study , 
Middle  States,  p.32) 

The  term  “outcomes”  is  used  extensively  in  the  2001  Middle  States  draft  standards  and  those  of  the 
Northwest  Association.  The  following  examples  illustrate  this  emphasis. 

• Northwest  (1999)  has  added  the  concept  of  evaluating  effectiveness  to  two  of  its  standards: 
“Standard  1.  Institutional  Mission  and  Goals,  Planning  and  Effectiveness”  and  “Standard  2. 
Educational  Program  and  Its  Effectiveness.”  It  also  has  a “Policy  on  Educational  Assessment”  that  is 


quite  specific  with  examples  of  outcomes  measures  which,  “when  used  in  appropriate  combinations 
and  informed  by  the  institutional  mission,  could  yield  an  efficacious  program  of  outcomes 
assessment.”  The  explanation  for  the  intent  of  this  policy  confirms  its  commitment  to  outcomes 
assessment: 

“The  intent  of  Commission  policy  is  to  stress  outcomes  assessment  as  an  essential  part  of  the 
ongoing  institutional  self-study  and  accreditation  processes,  to  underline  the  necessity  for  each 
institution  to  formulate  a plan  which  provides  for  a series  of  outcomes  measures  that  are 
internally  consistent  and  in  accord  with  its  mission  and  structure,  and  finally,  to  provide  some 
examples  of  a variety  of  successful  plans  for  assessing  educational  outcomes.”  (2.2.  “Policy  on 
Educational  Assessment,”  1992) 

Middle  States  (February  2001  draft)  describes  the  principles  that  guided  the  development  of  the 
proposed  new  standards,  “these  standards  consistently  emphasize  student  learning  and  student 
learning  outcomes....”  In  the  standards  an  emphasis  on  student  learning  is  paired  with  less  emphasis 
on  specific  resources.  “The  increased  emphasis  on  information  literacy  expands  and  clarifies  the  role 
of  learning  materials  and  library  professionals,  who  will  continue,  with  faculty  and  others,  to 
develop  this  important  skill.”  (p.  2) 

WASC  established  this  principle  as  part  of  its  redesign  work  for  the  January  2001  revised  Standards: 
“Principle  3:  Greater  emphasis  is  needed  on  evidence  of  educational  effectiveness  and  student 
learning.”  (p.  3)  It  also  promoted  educational  effectiveness  as  a “core  commitment.”  The  new 
WASC  review  process  model  relies  on  portfolios  that  can  be  updated  for  subsequent  reviews, 
attempting  to  move  to  a “best  practice”  model.  Much  of  what  might  go  into  a portfolio  would  be 
performance-based,  looking  at  outcomes  of  student  learning  and  how  planning  and  institutional 
resources  have  been  used  to  advance  the  mission  and  goals. 

North  Central  (March  2001)  has  revised  substantially  “Chapter  Reference  A.  Assessment  of  Student 
Academic  Achievement:  Levels  of  Implementation”  in  its  Handbook  of  Accreditation  providing  an 
“important  new  tool  to  assist  institutions  in  understanding  and  strengthening  their  programs  for 
assessment  of  student  academic  achievement....”  The  three  “Levels  of  Implementation”  contain 
characteristics  associated  with  a particular  level  at  any  given  time.  These  levels  are  intended  to 
provide  markers  of  the  progress  institutions  have  made  in  developing  their  assessment  programs. 
Certainly,  these  levels  of  assessment  apply  to  university  library  information  literacy  programs. 

Even  the  New  England  Commission,  whose  1992  standards  have  not  yet  been  revised,  initiated  in 
fall  2001  an  experiment  in  strengthening  assessment  of  student  learning  outcomes  and  institutional 
effectiveness.  This  project  will  invite  institutions  to  create  assessment  portfolios  which  will  be 
composed  of  the  following  parts:  vision  for  assessment;  fact  book  of  indicators  of  “input,  process, 
and  outcome  variables  with  particular  emphasis  on  those  relating  to  teaching  and  learning  (e.g., 
student  use  of  library  per  semester);”  presentations  of  selected  assessment  efforts,  such  as  course, 
program,  and  institution  level  assessment  studies;  scholarly  reflections  from  faculty  and 
administrators  about  their  efforts  to  conduct  assessment  and  their  findings;  and  peer  and  external 
review. 


5.  Information  literacy  emphasized 

As  a specific  student  learning  outcome,  information  literacy  has  been  promoted  in  all  but  two  (SACS 
and  North  Central)  of  the  standards  that  have  been  revised  since  1998.  Moreover,  more  text  exists  about 
the  shared  role  of  the  library  and  teaching  faculty  relating  to  information  literacy  instruction  in  the 
section  of  the  standards  covering  the  educational  program.  In  fact,  for  the  first  time  there  are  three 
standards  documents  (WASC,  Middle  States,  and  Northwest)  that  include  information  literacy  as  a 
general  educational  requirement.  (In  Northwest’s  document,  the  phrasing  varies  but  implies  information 
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literacy:  “literacy  in  the  discourse  or  technology  appropriate  to  the  program  of  study,”  Standard  2C. 
Undergraduate  Program.) 

The  document  developed  and  endorsed  by  all  eight  of  the  regional  accrediting  commissions,  “Best 
Practices  for  Electronically  Offered  Degree  and  Certificate  Programs,”  also  includes  a statement  in  the 
“Student  Support”  section  that  says,  “4  c.  The  institution  recognizes  that  appropriate  services  must  be 
available  for  students  of  electronically  offered  programs....  With  variations  for  specific  situations  and 
programs,  these  services  may  include. ..training  in  information  literacy,  including  research 
techniques....” 

6.  Few  specific  references  to  electronic  and  networked  resources  and  services 
There  is  very  little  specific  text  in  the  standards  and  supporting  documents  referencing  electronic  and/or 
networked  resources  and  services.  There  are  only  five  instances  of  “network”  or  “electronic  resources 
and  services,”  or  equivalent  terms  like  “electronic  bibliographic  databases”  or  “online  catalogs.”  Here 
are  the  extracts  identified: 

Northwest:  5.  A.  Purpose  and  Scope.  Three  statements  defining  the  components  of  information  resources 
and  services  “(e.g.,  networks,  telecommunication  facilities,  computer  centers,  libraries,  media  center); 
and  the  info,  resources/services  supporting  the  educational  programs.” 

SACS  (1998):  5.1.2.  Services.  “Convenient,  effective  access  to  electronic  bibliographic  databases, 
whether  on-site  or  remote,  must  be  provided  when  necessary  to  support  the  academic  programs.” 

SACS  (1998):  5.1.3.  Information  Technology  Resources  and  Systems.  “A  reliable  data  network  should 
be  available  so  that  students,  faculty,  and  staff  may  become  accustomed  to  electronic  communication 
and  familiar  with  accessing  national  and  global  information  resources.” 

North  Central:  Criterion  2.  Human,  Financial,  and  Physical  Resources,  “j.  Academic  resources  and 
equipment  (e.g.,  libraries,  electronic  services  and  products,  learning  resource  centers...)  are  adequate  to 
support  the  institution’s  purposes.” 

“Best  Practices  for  Electronically  Offered  Degree  and  Certificate  Programs.”  4.  Student  Support.  4.c. 
“The  institution  recognizes  that  appropriate  services  must  be  available  for  students  of  electronically 
offered  programs....  With  variations  for  specific  situations,  these  services  may  include:  ...remote  access 
to  data  bases,  online  journals  and  full-text  resources,  document  delivery  services....” 

However,  there  are  many  references  to  the  phrase  “information  resources  and  services,”  “information 
technology  resources  and  services”  or  “information  technologies.”  I suggest  that  these  broader  phrases 
are  intended  to  include  electronic  and  networked  resources  and  services.  This  lack  of  explicit  references 
that  use  the  precise  phrase  “electronic  and  networked  resources  and  services”  doesn’t  mean  that  the 
accrediting  commissions  don’t  expect  libraries  to  demonstrate  how  these  resources  and  services  support 
student  learning  and  faculty  research.  While  there  may  not  be  a need  to  demonstrate  the  outcomes  of 
electronic  and  networked  resources  and  services,  there  is  a need  to  provide  evidence  that  documents  the 
connections  between  their  use  and  fulfillment  of  the  institution’s  goals.  What  follows  are  but  four 
examples  that  illustrate  this  connection: 
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Northwest:  Standard  2.  Educational  Program.  2.A.8.  “Faculty,  in  partnership  with  library  and 
information  resources  personnel,  ensure  that  the  use  of  library  and  information  resources  is  integrated 
into  the  learning  process.” 

New  England:  Standard  7.  Library  and  Information  Resources.  7.1.  “...The  institution  ensures  that 
students  use  these  resources  as  an  integral  part  of  their  education.” 

WASC:  Standard  2.  Achieving  Educational  Objectives  through  Core  Functions,  subsection  Teaching 
and  Learning.  Criterion  2.3.  “The  institution’s  expectations  for  learning  and  student  attainment  are 
clearly  reflected  in  its  academic  program  and  policies.  These  include  the  organization  and  content  of  the 
curricula.. .and  the  use  of  its  library  and  information  resources.” 

WASC:  Eligibility  Requirements  #17.  “...The  institution  must  also  be  able  to  demonstrate  that  library 
use  is  a fundamental  part  of  curricula.” 

Throughout  this  report,  then,  the  phrase  “information  resources  and  services”  is  understood  to  include 
electronic  and  networked  resources  and  services. 

III.  Comparison  of  accreditation  standards’  treatment  of  outcomes 

Figure  1 provides  text  from  the  standards  and  appropriate  supporting  documentation  related  to  outcomes 
and  impacts  that  reference  or  imply  the  contributions  of  university  libraries.  Text  that  uses  equivalent 
terms  like  “competencies”  and  phrases  such  as  “achievement  of  educational  goals  and/or  learning 
objectives,”  “evaluation  of  student  performance,”  and  “evidence  of  educational  effectiveness  and 
student  learning”  is  included.  These  phrases  imply  outcomes  that  are  either  explicitly  or  less  directly 
connected  to  the  contributions  of  university  libraries. 

Statements  were  selected  that  show  how  outcomes  are  articulated  and  conceptualized  with  a particular, 
but  not  exclusive,  focus  on  libraries  and  electronic  and  networked  services.  The  context  of  these 
excerpts  is  identified  by  the  name  and/or  section  of  the  standard.  In  cases  where  the  connections  are  less 
obvious  or  direct,  parenthetical  comments  suggest  how  the  statement  might  apply  to  university  libraries. 
Some  of  the  less  direct  items  are  included  because  the  text  refers  to  “all  programs  or  units.”  In  a few 
cases,  the  statement  may  not  refer  strictly  to  outcomes,  but  rather  to  the  impact  or  effects  of  library  and 
information  resources  and  services.  The  following  observations  can  be  drawn  from  the  extracts. 

1 .  The  majority  of  these  outcomes  and  outcomes-related  statements  that  refer  to  libraries  and 
information  resources  are  located  in  sections  of  the  standards  that  deal  with  the  education  program  and 
institutional  effectiveness. 

2.  The  use  of  library  and  information  resources  is  connected  to  student  learning  outcomes  in  four  of  the 
documents,  and  evidence  such  as  inclusion  in  course  syllabi  and  integration  of  library  use  into  the 
undergraduate  curriculum  are  offered  as  measurable  indicators  for  assessment  purposes  in  two  of  the 
documents. 

3.  The  university  library’s  role  in  helping  students  develop  information  literacy  skills  is  an  important 
student  learning  outcome  referenced  in  four  of  the  documents  and  in  the  “Best  Practices  for 
Electronically  Offered  Degree  and  Certificate  Programs”  endorsed  by  the  accrediting  commissions. 
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4.  Assessing  student  needs,  perceptions,  and  levels  of  satisfaction  with  educational  support  services  (i.e., 
library  and  information  services)  and  demonstrating  that  the  findings  from  these  user  studies  are  used 
for  program  improvement  is  a fundamental  expectation  of  all  the  regional  accrediting  commissions. 

5.  Appraisal  of  annual  institutional  goals  and  progress  in  their  accomplishment  are  suggested  as  a type 
of  evidence  contributing  to  institutional  outcomes,  or  in  some  of  the  documents  the  phrase  used  is 
“institutional  effectiveness.” 

6.  All  of  the  standards  describe  the  need  for  institutions  to  have  an  assessment  or  evaluation  plan  and  to 
document  that  the  findings  are  utilized  for  program  improvement.  Some  of  the  documents  clarify  this 
requirement  to  mean  that  each  program  or  unit  should  have  an  assessment  plan. 

7.  Several  of  the  documents  refer  to  the  campus  climate  or  the  institutional  environment  that  supports 
teaching  and  learning.  Three  specifically  connect  library  and  information  resources  and  services  to  the 
quality  of  the  learning  environment.  The  implication  is  that  university  libraries  should  clearly  describe 
the  resources  and  services  they  provide  that  directly  support  the  learning  environment,  how  these  are 
used,  and  with  what  effects  on  students  and  faculty. 
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Figure  1.  Text  Related  to  Outcomes  and  Impacts  Directly  and  Indirectly  Involving  Libraries  and 

Information  Resources  and  Services 


Western  Association  of  Schools  and  Colleges  (January  2001) 

Standard  2.  Achieving  Educational  Objectives  through  Core  Functions,  subsection  “Teaching  and 
Learning.” 

Criterion  2.2  Baccalaureate  programs  engage  students  in  an  integrated  course  of  study.. ..These  programs  also  ensure  the 
development  of  core  learning  abilities  and  competencies  including,  but  not  limited  to,  college-level  written  and  oral 
communication,  quantitative  skills,  information  literacy.... 

2.2  Guideline.  Competencies  required  for  graduation  are  reflected  in  course  syllabi  for  both  general  education  and  the  major. 

Criterion  2.3.  The  institution’s  expectations  for  learning  and  student  attainment  are  clearly  reflected  in  its  academic  program 
and  policies.  These  include  the  organization  and  content  of  the  curricula... and  the  use  of  its  library  and  information 
resources. 

2.3  Guideline.  The  use  of  information  and  learning  resources  beyond  textbooks  is  evidenced  in  syllabi  throughout  the 
undergraduate  and  graduate  curriculum. 

Criterion  2.7.  In  order  to  improve  program  currency  and  effectiveness,  all  programs  offered  by  the  institution  are  subject  to 
review,  including  analyses  of  the  achievement  of  the  program’s  learning  objectives  and  outcomes.  (A  university  library’s 
information  literacy  program  clearly  applies.  To  the  extent  that  libraries  connect  their  educational  programming  to  the 
curriculum,  it  is  also  possible  to  consider  under  this  criterion  a university  library’s  exhibits  program,  lecture/film  series,  or 
other  similar  programming  that  is  directly  linked  to  the  curriculum.) 

(The  following  are  relevant  “Questions  for  Institutional  Engagement”  provided  in  the  WASC  document  to  generate  measures 
and  qualitative  documentation  to  demonstrate  how  the  library  directly  contributes  to  these  three  criteria.) 

Questions  for  Institutional  Engagement  3.  To  what  extent  does  the  institution  provide  an  environment  that  is  actively 
conducive  to  study  and  learning,  where  library,  information  resources,  and  co-curricular  programs  actively  support  student 
learning? 

Questions  for  Institutional  Engagement  8.  To  what  extent  does  the  institution  ensure  that  students  develop  expected  core 
learning  abilities  and  competencies  before  they  graduate?  (Applies  to  information  literacy) 

Standard  2.  subsection  “Support  for  Student  Learning.” 

(Even  though  the  library  is  not  named  here,  as  a major  instructional  and  instructional  support  program,  an  expectation  for  its 
involvement  and  its  impact  is  implied.) 

Criterion  2.10.  Regardless  of  mode  of  program  delivery,  the  institution  regularly  identifies  the  characteristics  of  its  students 
and  assesses  their  needs,  experiences,  and  levels  of  satisfaction.  The  information  is  used  to  help  shape  a learning-centered 
environment  and  to  actively  promote  student  success. 

Question  for  Institutional  Engagement  6.  In  what  ways  does  the  institution  gather,  analyze  and  use  information  about  the 
needs  and  preferences  of  students  and  the  values  they  place  on  programs  and  services?  Is  this  information  effectively  used  to 
create  a climate  conducive  to  student  and  institutional  learning? 
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Southern  Association  of  Colleges  and  Schools  (1998) 


Section  V.  Educational  Support  Services. 

It  is  expected  that  each  program  or  unit  will  establish  goals  which  derive  from  and  support  the  purposes  of  the  institution, 
evaluate  its  success  in  achieving  these  goals  and  demonstrate  the  use  of  the  evaluation  in  making  appropriate  modifications 
in  resources,  programs  and  services. 

Distance  Education:  Definition  And  Principles:  Policy  Statement,  subsection  “Library  and 
Learning  Resources.”  (1997,  updated  May  2000) 

The  institution  ensures  that  students  have  access  to  and  can  effectively  use  appropriate  library  resources.  The  institution 
monitors  whether  students  make  appropriate  use  of  learning  resources. 

Section  III.  Institutional  Mission,  Governance  and  Effectiveness,  subsection  “Institutional 
Effectiveness.”  (March  2001  Draft) 

14.  The  institution  identifies  expected  outcomes  for  its  educational  programs  and  its  administrative  and  educational  support 
services;  assesses  whether  it  achieves  these  outcomes  and  provides  evidence  of  improvement  based  on  analysis  of  those 
results. 

Section  IV.  Programs,  subsection  “Educational  Programs.” 

1.  The  institution  demonstrates  that  each  educational  program  for  which  academic  credit  is  awarded. ..has  stated  program  and 
learning  outcomes  and  ensures  appropriate  levels  of  student  achievement.  (Applies  to  information  literacy  credit  courses 
and/or  information  literacy  components  within  other  courses.) 

14.  The  institution’s  use  of  technology  enhances  student  learning,  is  appropriate  for  meeting  the  objectives  of  its  programs, 
and  ensures  that  students  have  access  to  and  training  in  the  use  of  technology.  (Indirect  but  can  apply  to  electronic  and 
networked  resources  and  services.) 


Northwest  Association  of  Schools  and  Colleges  (1999) 

Standard  l.B.  Planning  and  Effectiveness.  Required  Supporting  Documentation. 

Evidence  that  demonstrates  the  analysis  and  appraisal  of  institutional  outcomes;  such  as:  annual  goals  and  assessment  of 
success  in  their  accomplishments;  studies  regarding  effectiveness  of  programs  and  their  graduates;  test  comparisons  that 
reveal  beginning  and  ending  competencies;  satisfaction  surveys  of  students,  alumni  and  employees.  (University  libraries 
would  be  included  and  could  use  as  evidence  annual  goals  and  assessment  of  success  in  their  accomplishments  and 
satisfaction  surveys.) 

Standard  2.  Educational  Program.  2.A.3. 

Degree  and  certificate  programs  demonstrate  a coherent  design. ..synthesis  of  learning,  and  the  assessment  of  learning 
outcomes;  and  require  the  use  of  library  and  other  information  resources. 

Standard  2.B.2. 

The  institution  identifies  and  publishes  the  expected  learning  outcomes  for  each  of  its  degree  and  certificate  programs. 
Through  regular  and  systematic  assessment  it  demonstrates  that  students  who  complete  their  programs,  no  matter  where  or 
how  offered,  have  achieved  these  outcomes.  (Programs  that  have  integrated  information  literacy  skills  are  implied.) 

Standard  2.C.  Undergraduate  Program. 

Baccalaureate... programs  include  a substantial  core  of  general  education  instruction  with  identifiable  outcomes  and  require 
competence  in  a.  written  and  oral  communication;  b.  quantitative  reasoning,  c.  critical  analysis  and  logical  thinking,  and  d. 
literacy  in  the  discourse  or  technology  appropriate  to  the  program  of  study.  (Implied  is  the  effective  use  of  electronic  and 
networked  resources.) 


ERIC 
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2.2.  Policy  on  Educational  Assessment. 

The  Commission  expects  each  institution  and  program  to  adopt  an  assessment  plan  responsive  to  its  mission  and  its 
need....  The  Commission  urges  the  necessity  of  a continuing  process  of  academic  planning,  the  carrying  out  of  those  plans, 
the  assessment  of  the  outcomes,  and  the  influencing  of  the  planning  process  by  the  assessment  activities.  (Applies  to 
university  libraries.) 

2.6.  Policy  on  Distance  Delivery  of  Courses,  Certificate,  and  Degree  Programs. 

2.  The  institution  evaluates  the  educational  effectiveness  of  its  distance  education  programs  (including  assessments  of 
student  learning  outcomes,  student  retention,  and  student  satisfaction.  (Implied  is  that  student  satisfaction  with  library  and 
electronic  resources  and  services  would  be  included  in  student  surveys.) 

Required  Supporting  Documentation  for  Standard  2. 

2.  Inventory  of  documents  that  demonstrate  the  appraisal  of  educational  program  outcomes.  Examples  may  include:  annual 
goals  and  assessment  of  success  in  their  accomplishment;  test  comparisons  that  reveal  beginning  and  ending  competencies; 
surveys  of  student  satisfaction. 

Standard  3.  Students.  Supporting  Documentation.  Required  Exhibits. 

7.  Evidence  of  the  impact  of  student  services  on  students.  (Clearly  the  university  library  could  connect  use  and  user 
satisfaction  measures  from  some  of  its  services,  such  as  those  specifically  listed  in  the  “Supporting  Documentation  for 
Standard  5”  below.) 

Standard  4.  Faculty.  Required  Supporting  Documentation  for  Standard  4. 

Representative  examples  of  the  institutional  and  public  impact  of  faculty  scholarship....  (This  could  apply  to  librarian- 
scholars  as  well  as  university  library  direct  support  to  faculty  scholarship  and  research.) 

Standard  5.E.  Library  and  Information  Resources.  Planning,  and  Evaluation. 

Library  and  information  resources  planning  activities  support  teaching  and  learning  functions  by  facilitating  the  research  and 
scholarship  of  students  and  faculty.  Related  evaluation  processes  regularly  assess  the  quality,  accessibility  and  use  of 
libraries  and  other  information  resource  repositories  and  their  services  to  determine  the  level  of  effectiveness  in  support  of 
the  educational  program. 

5.E.3.  The  institution  regularly  and  systematically  evaluates  the  quality,  adequacy  and  utilization  of  its  library  and 
information  resources  and  services.... 

Supporting  Documentation  for  Standard  5. 

Fourteen  “required  exhibits”  including:  policies,  student  and  faculty  orientation  materials,  use  statistics,  collection  statistics, 
assessment  measures  to  determine  the  adequacy  of  holdings  and  services  to  support  the  educational  programs  both  on  and  off 
campus,  computer  usage  statistics  related  to  retrieval  of  library  resources,  comprehensive  budgets  for  library  and  information 
resources. 


New  England  Association  of  Schools  and  Colleges  (1992) 

(Perhaps  because  this  association’s  standards  date  to  1992,  none  of  the  standards  refer  to  outcomes  or  use  equivalent 
language.  However,  a pilot  institutional  assessment  project  is  underway  “to  encourage  its  member  institutions  to  strengthen 
the  assessment  of  student  learning  and  institutional  effectiveness.”  The  pilot  will  select  10  institutions  for  fall  2001  and 
continue  for  two  years.  The  project  involves  the  use  of  institutional  assessment  portfolios  that  are  to  include  the  following 
components.) 

1.  vision  for  assessment-a  description  of  the  institution’s  vision  for  assessment...  of  expectations  and  available  resources  in 
relation  to  the  assessment  of  student  learning  outcomes  and  institutional  effectiveness. 
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2.  fact  book  indicators-input,  process,  and  outcomes  variables  with  particular  emphasis  on  those  related  to  teaching  and 
learning. ..(e.g.,  students  involved  with  faculty  in  research, ...student  use  of  library  per  semester,  and  hours  of  study  per  week. 

3.  presentations  of  selected  assessment  efforts-a  selection  of  what  the  institution  regards  as  some  of  the  most  recent  and 
useful  course,  program,  and  institution-level  assessment  efforts  would  be  presented.  Another  example  might  be  a description 
of  the  implementation  and  findings  regarding  student  outcomes  from  a capstone  course  designed  to  understand  to  what 
degree  students  have  developed  the  desired  competencies  expected  within  a given  program. 

4.  scholarly  reflections  from  faculty  and  administrators-selected  faculty  and  administrative  leaders  would  present  scholarly 
reflections  of  their  efforts  to  conduct  assessment  and  inquiry.... 

5.  peer  and  external  review-the  portfolio  would  present  reviews  by  peers  regarding  their  sense  of  what  the  institution  is 
accomplishing  with  its  assessment  efforts.... 

(This  pilot  can  provide  participating  university  libraries  opportunities  to  help  identify  which  measures  and  other 
documentation  will  be  used  to  show  evidence  of  their  contributions  and  impact  on  student  learning  and  other  institutional 
goals.) 


North  Central  Association  of  Colleges  and  Schools  (March  2001) 

(While  there  are  no  explicit  references  to  outcomes  and  libraries,  one  standard  contains  text  that  implies  library 
contributions.) 

Standard  3. 

The  institution  is  accomplishing  its  educational  and  other  purposes. ...The  Commission  considers  evidence  such  as: 

b.  assessment  of  appropriate  student  academic  achievement  in  all  its  programs,  documenting  proficiency  in  skills  and 
competencies  essential  for  all  college-educated  adults. 

c.  graduate  programs  that  expect  students  and  faculty  to  value  and  engage  in  research,  scholarship  and  creative  activity. 

(The  effects  of  information  literacy  programs  and  librarian-departmental  teaching  faculty  collaborations  can  be  implied  in  the 
two  types  of  “evidence”  described  in  3b.  and  3c.) 

(North  Central  has  a very  detailed  document,  “Chapter  Reference  A.  Assessment  of  Student  Academic  Achievement:  Levels 
of  Implementation”  (updated  March  1,  2001)  in  its  Handbook  of  Accreditation.  The  “Levels  of  Implementation”  are  intended 
“to  assist  institutions  in  understanding  and  strengthening  their  programs  for  assessment  of  student  academic  achievement....” 
This  document  contains  three  levels  from  level  1 “beginning  implementation  of  assessment  programs”  to  level  3 “maturing 
stages  of  continuous  improvement.”  The  clusters  of  characteristics  contained  in  these  levels  serve  as  indicators  of  progress 
on  a continuum  of  assessment.  None  of  the  statements  in  this  document  refer  specifically  to  libraries  and  information 
resources,  but  implied  is  the  library’s  role  in  teaching  information  literacy  as  a competency  “essential  for  all  college-educated 
adults.” 

North  Central’s  redesign  of  its  accreditation  process  has  resulted  in  “the  Commission’s  new  alternative  process  for 
maintaining  accreditation,  the  Academic  Quality  Improvement  Project”  <http://www.AQIP.org/criteria.html>.  It  appears 
very  useful  for  outcomes  assessment,  as  each  major  criterion  has  recommended  questions  to  provoke  measures  of 
effectiveness  and  outcomes.) 


Middle  States  Association  of  Colleges  and  Schools  (1994) 

(The  standard  Institutional  Effectiveness  and  Outcomes  contains  about  one  page  of  text.  Following  are  pertinent  statements.) 

Institutional  Effectiveness  and  Outcomes. 
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Outcomes  assessment  involves  gathering  and  evaluating  both  quantitative  and  qualitative  data  which  demonstrate 
congruence  between  the  institution’s  mission,  goals,  and  objectives  and  the  actual  outcomes  of  its  educational  programs  and 
activities.  The  ultimate  goal  of  outcomes  assessment  is  the  improvement  of  teaching  and  learning.  The  approaches  may  vary 
and  need  not  be  elaborate  or  dependent  on  quantitative  criteria,  but  they  should  be  systematic  and  thorough....  An  institution 
should  be  able  to  demonstrate  that  the  information  obtained  is  used  as  a basis  for  ongoing  self-renewal.  In  addition  to 
assessing  academic  achievement,  institutions  should  seek  ways  to  assess  the  degree  to  which  students’  attitudes,  social  and 
ethical  values,  interests,  and  commitment  to  scholarship  and  lifelong  learning  develop  as  a result  of  their 
education... .Especially  significant  are  institutional  studies  which  provide  insights  into  effective  teaching  and  the  role  of 
campus  climate  in  promoting  student  learning  and  development.”  (University  libraries’  impact  on  campus  climate  is 
implied.) 

Standard:  Library  and  Learning  Resources. 

Paragraph  #7:  Librarians,  information  specialists,  and  other  staff  must  demonstrate  their  professional  competence  on  the 
basis  of  criteria  comparable  to  those  for  other  faculty  and  staff.  They  should  also  help  facilitate  the  teaching  and  learning 
process,  especially  in  assisting  students  to  improve  their  information  skills.... 

Paragraph  # 9:  A system  for  assessing  the  effectiveness  of  library  and  learning  resource  should  be  available.  It  should  focus 
on  utilization,  accessibility,  availability  and  delivery  of  materials. 

Framework  for  Outcomes  Assessment.  (1996) 

(This  separate  document,  presents  guidance  for  institutions  preparing  to  meet  the  outcomes  assessment  requirements  as 
delineated  in  the  standards  Principles,  process  and  methods  are  discussed.  Selected  statements  follow.) 

Student  cognitive  development  should  be  one  of  several  components.  Other  measures  might  include  student  evaluations  of 
instructional  quality,  measures  of  student  satisfaction  with  the  quality  of  support  services..  ..(p.  14) 

The  analysis  of  student  achievement  with  respect  to  general  education  utilizes  different  measurement  objectives  of  assessing 
competencies  in  4 broad  areas:  cognitive  abilities. ..competence  in  information  management  skills  and  communication.... (p. 


Learning  Environments-Assessments  of  program  majors  typically  extend  beyond  examining  learning  outcomes  to  include,  as 
additional  indicators  of  program  quality,  various  aspects  of  the  educational  environment  which  are  related  to  or  affect  student 
learning  in  the  major.. ..Examples  of  such  measures,  listed  in  Figure  4,  relate  to  program  inputs  and  processes.  While  they 
offer  no  direct  information  on  the  extent  and  quality  of  student  learning,  they  can  provide  indirect  information  on  educational 
quality  when  their  relationship  to  the  breadth  and  depth  of  student  learning  has  been  determined,  (p.  21) 

(Figure  4 shows  “student  use  of  learning  resources”  as  a process  measure  and  “library  holdings”  as  an  input  measure.  The 
implication  expressed  here  is  that  specific  library  and  electronic  resources  usage  measures,  in  conjunction  with  descriptive 
information  about  collections  and  access  to  resources,  are  important  “indirect  information”  about  the  quality  of  the  learning 
environment.) 

Middle  States  Association  of  Colleges  and  Schools  (February  2001  Draft) 

Standard  1.  Mission,  Goals  and  Objectives. 

Objectives  are  outcomes-based  and  measurable,  and  thus  they  provide  mechanisms  for  on-going  review  and  refinement  of 
goals. 
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Standard  4.  Institutional  Resources. 


The  effective  and  efficient  use  of  the  institution’s  resources  is  analyzed  as  part  of  ongoing  outcomes  assessment. 

(In  the  “Context”  narrative  it  is  made  clear  that  this  standard  includes  “technology,  research,  and  instructional  support 
resources.”) 

Supplemental  Analysis  and  Documentation. 

...although  not  required,  the  following  may  facilitate  the  institution’s  own  analysis  relative  to  this  accreditation  standard: 
evidence  of  cooperative  agreements  for  inter-institutional  collaboration  and  resource  sharing,  analyses  of  any  resulting 
efficiencies,  and  impact  on  student  achievement  of  academic  goals. 

Standard  10.  Educational  Program. 

The  institution’s  education  programs  display  academic  content,  rigor  and  coherence.... The  institution  identifies  student 
learning  outcomes,  including  knowledge,  skills  and  values  for  each  program. 

(Within  the  “Context”  section  for  Standard  10  is  found  this  statement  about  information  literacy:  “Information  literacy... is 
vital  to  all  disciplines  and  to  effective  teaching  and  learning  in  any  institution.. .Academic  quality,  student  learning,  teaching 
performance,  and  institution  effectiveness  are  enhanced  by  programs  that  assist  students,  faculty  and  instructional  staff  to  use 
information  resources  in  a variety  of  media  and  formats.”) 

“Fundamental  Elemental  of  Educational  Program.” 

Learning  resources,  library  services,  and  professional  library  staff  support  adequate  to  support  the  institution’s  educational 
programs;  collaboration  between  professional  library  staff  and  faculty  in  teaching  and  fostering  information  literacy  skills 
relevant  to  the  curriculum  and  to  faculty  research;  programs  that  promote  student  use  of  information  and  learning  resources. 

Supplemental  Analysis  and  Documentation. 

. . .evidence  of  course  syllabi,  incorporating  learning  outcomes;  review  of  results  from  the  institution’s  implemented 
outcomes  assessment  plan;  assessment  of  information  literacy  outcomes;. ..evidence  of  information  literacy  incorporated  in 
the  curriculum  with  syllabi.... 

(The  following  types  of  evidence,  while  not  outcomes,  are  included  in  this  section.) 

...evidence  of  local  and  remote  information  resources,  access  structures,  and  technologies  for  experimentation  adequate  to 
support  the  curriculum;  evidence  of  accessible  reference  tools  to  ascertain  where  relevant  materials  exist  and  are  located; 
evidence  of  trained  instructional  and  reference  staff  or  other  support  services,  available  on-site  or  via  remote  access,  to  help 
students  and  teaching  staff  locate  and  evaluate  information  tools  and  resources;  evidence  of  an  adequate  policy  and  process 
for  the  development  and  management  of  information  resources  tailored  to  the  mission  and  goals  of  the  institution. 

(It  is  implied  that  these  types  of  input  measures  can  provide  partial  evidence  of  a quality  learning  environment.) 

Standard  11.  General  Education.  “Fundamental  Elements  of  General  Education.” 

Relative  to  this  standard,  an  accredited  institution  is  characterized  by:  general  education  requirements  that  assure  that  upon 
degree  program  completion,  students  are  proficient  in  oral,  written  and  performance  communication;  scientific  and 
quantitative  reasoning,  technological  competency,  and  information  literacy  which  includes  critical  analysis  and  reasoning; 
assessment  of  general  education  outcomes  within  the  institution’s  overall  plan  for  assessing  student  learning. 

Standard  13.  Institutional  Assessment. 

The  institution  has  developed  and  implemented  an  assessment  plan...  and  its  effectiveness  in  assuring  that  its  students  and 
graduates  achieve  the  appropriate  learning  and  other  outcomes.... While  the  Commission  expects  institutions  to  engage  in 
outcomes  assessment,  it  does  not  prescribe  a specific  approach  or  methodology. ...Nevertheless,  an  institution  engaged  in  self- 
study  or  periodic  review  should  provide  evidence  that  the  assessment  of  outcomes,  particularly  learning  outcomes,  is  an 
ongoing  institutional  activity. 

“Fundamental  Elements  of  Institutional  Assessment.” 
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...a  written  assessment  plan  and  process  that  meets  the  following  criteria:  ...assesses  periodically  the  achievement  of 
institutional  goals  that  include  the  total  range  of  curricula,  activities,  and  support  services;. ..systematic  and  thorough 
multiple  qualitative  and  quantitative  measures,  which  maximize  the  use  of  existing  data  and  information.... 
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Standard  14.  Assessment  of  Student  Learning. 

Although  related  to  overall  institutional  effectiveness,  the  assessment  of  student  learning  outcomes  has  the  student  as  its 
primary  focus  of  inquiry....  Outcomes  assessment  is  essential  regardless  of  the  nature  of  the  institution,  its  particular  mission, 
the  types  of  programs  it  offers,  or  the  manner  in  which  its  educational  programs  are  delivered  and  student  learning  facilitated. 

“Fundamental  Elements  of  Assessment  of  Student  Learning.” 

...learning  objectives  for  individual  courses  and  for  all  programs;... an  implemented  plan  for  assessing  all  educational 
programs,  including  basic  skills  programs,  regardless  of  where  or  how  delivered,  that  includes  multiple  measures  of  student 
learning,  at  least  some  of  which  directly  address  learning  outcomes. 

Supplemental  Analysis  and  Documentation. 

Evidence  of  assessment  plans  that  include  multiple  types  of  outcomes,  such  as  cumulative  learning,  analytical  and 
information  skills.... 


Best  Practices  for  Electronically  Offered  Degree  and  Certificate  Programs.  (2001) 

developed  and  endorsed  by  the  eight  regional  accrediting  commissions 

(The  following  selected  statements  are  considered  essential  to  quality  distance  education  programming.) 

Curriculum  and  Instruction  section. 

Methods  change  but  standards  of  quality  endure.  The  important  issues  are  not  technical  but  curriculum-driven  and 
pedagogical.  Decisions  about  such  matters  are  made  by  qualified  professionals  and  focus  on  learning  outcomes  for  an 
increasingly  diverse  student  population. 

Question  for  Review. 

Does  the  program  design  involve  the  demonstration  of  such  skills  as  analysis,  comprehension,  communication,  and  effective 
research? 

Evaluation  and  Assessment  section. 

5d.  Overall  program  effectiveness  is  determined  by  such  measures  as.. .student  satisfaction  as  measured  by  regular 
surveys.. .the  extent  to  which  access  is  provided  to  students  not  previously  served. ..measures  of  the  extent  to  which  library 
and  learning  resources  are  used  appropriately  by  the  program’s  students.. .usage  records  concerning  use  of  the  library  and 
learning  resources  and  instructor  assignments  that  require  such  usage. 
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IV.  Concluding  Observations  and  Recommendations 

Based  on  the  content  analyses,  especially  the  text  in  Figures  1 and  2,  the  following  observations  and 
recommendations  are  made: 

1.  The  university  library,  in  collaboration  with  faculty,  plays  a major  role  as  an  academic  program  in 
teaching  information  literacy  skills.  Probably  the  most  direct  contribution  the  library  makes  to 
institutional  goals  is  its  role  in  developing  clear  student  learning  outcomes  for  information  literacy 
skills;  assessing  the  progress  and  achievement  of  these  outcomes;  and  showing  how  the  findings  are 
used  to  improve  student  learning.  For  those  institutions  where  information  literacy  credit  courses,  or 
components  of  credit  courses,  are  offered  and  where  information  literacy  skills  are  part  of  general 
education  requirements,  this  contribution  is  even  stronger. 

2.  The  university  library  is  also  an  important  academic  support  unit  and  as  such  provides  measurable 
information  services  and  resources  that  contribute  to  student  learning.  This  observation  is  buttressed  by 
the  fact  that  more  of  the  regional  standards  have  included  references  to  the  use  of  libraries  and 
information  resources  and  services  in  the  “Educational  Program”  section  of  the  standards,  thus 
connecting  it  directly  to  teaching  and  learning  standards.  In  fact,  four  of  the  six  either  require  use  of 
library  and  information  resources  or  use  language  like  “use  of  library  and  information  resources  is 
fundamental.”  These  contributions  are  articulated  or  implied  in  several  ways: 

a.  by  suggesting  that  course  syllabi  or  course  learning  outcomes  be  used  as  sources  of  data  to 
document  usage  of  library  and  information  resources; 

b.  by  providing  services,  exhibits,  and  programming  that  directly  contributes  to  physical  and  virtual 
campus  environments  that  are  conducive  to  learning  and  study; 

c.  by  offering  specific  services  and  resources  to  graduate  programs  and  distance  education  programs. 
University  libraries  have  an  opportunity  to  present  various  quantitative  and  qualitative  measures  that 
demonstrate  how  the  library  supports  graduate  programs  and  distance-learning  courses  and  degree 
programs.  While  a university  library’s  support  may  not  be  that  different  than  how  it  supports  on- 
campus  learners,  the  clustering  and  presentation  of  multiple  measures  that  directly  support  distance 
learners  are  evidence  of  the  library’s  instructional  support  role  to  a potentially  important  and 
growing  segment  of  the  university; 

d.  by  providing  resources  and  services  to  support  faculty  research,  scholarship,  and  professional 
development;  and 

e.  by  collaborating  with  other  information  technology  providers  to  plan  for  and  evaluate  the  adequacy 
and  access  to  information  resources  and  services  provided  online. 

3.  University  libraries  need  to  make  more  explicit  and  public  the  connections  between  the  following: 

a.  how  their  resources  and  services  support  institutional  goals.  In  some  of  the  standards,  appraisal  of 
annual  institutional  goals  and  progress  in  their  accomplishment  is  suggested  as  a type  of  evidence 
contributing  to  institutional  effectiveness; 

b.  how  their  resources  and  services  are  used,  by  whom,  and  the  effects  or  impacts  of  this  use;  and 

c.  how  their  strategic  plans  and  assessment  plans  support  the  institution’s  planning  documents  and 
assessment  process  and  how  the  findings  of  the  library’s  assessment  activities  contribute  to  the 
achievement  of  the  institution’s  mission  and  goals. 

4.  Perhaps  university  libraries  should  reserve  the  term  “outcomes”  for  those  measures  and  supporting 
documentation  that  provide  evidence  of  student  performance  and  proficiency  related  to  information 
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literacy  skills,  while  using  the  term  “impacts”  or  “effects”  to  document  their  other  contributions  to 
institutional  goals.  This  suggestion  is  based  on  the  text  of  several  of  the  standards  which  refer  to  the 
library’s  contributions  to  institutional  or  educational  effectiveness.  There  is  little  specific  guidance  or 
requirements  about  how  to  demonstrate  the  library’s  contributions  to  institutional  effectiveness  in  the 
standards  that  were  reviewed.  However,  a main  point  made  in  all  the  standards  is  that  each  program  and 
unit  of  the  institution  is  to  evaluate  its  contributions  and  effectiveness  against  the  goals  it  sets  for  itself. 
Of  course,  these  goals  are  directly  linked  to  the  university  mission  and  goals.  University  libraries  could 
strengthen  their  evidence  by  triangulating  data  and  supporting  information.  They  could  group  such 
measures  as  selective  qualitative  descriptions  of  resources  and  services  with  usage  data  for  specific 
services  and  with  user  satisfaction  and  perceptions  of  benefit  data  findings.  All  of  this  could  be  grouped 
to  show  the  impact  of  library  and  information  resources  and  services  on  the  contributions  listed  above  in 
#2.  a.-e.  Wherever  possible,  this  type  of  clustering  could  be  organized  by  academic  programs  or 
schools. 
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Figure  2.  Comparison  of  Documents  with  and  without  a Separate  Standard  Relating  to  Libraries 

and  Information  Resources 

Below  is  a comparison  of  documents  that  currently  have  a separate  standard  for  libraries  and 
information  resources  with  those  that  do  not.  The  first  four  entries  (New  England,  Northwest,  Southern 
1998,  and  Middle  States  1994)  compare  the  text  of  standards  and  other  relevant  documents  that  have  a 
separate  standard  for  libraries  and  information  resources.  The  last  four  entries  (North  Central,  Southern 
2001,  WASC,  and  Middle  States  2001),  below  the  double  line,  compare  the  text  of  standards  and  other 
relevant  documents  for  those  accrediting  commissions  that  lack  a separate  standard  for  libraries  in  their 
current  standards  or  in  the  proposed  revisions. 


Regional  Commission 

Components  and  Summary  of  Content  for  Separate  Standard(s)  Relating  to 
Libraries 

New  England  (1992) 

Standard  7.  Library  and  Information  Resources 

Six  paragraphs  summarized  below. 

7. 1 Covers  availability  of  suitable  resources  to  support  instruction.  “The  institution  ensures 
that  students  use  these  resources  as  an  integral  part  of  their  education.” 

7.2  Deals  with  ownership  or  guaranteed  access  of  collections  and  services  and  provision  of 
facilities  adequate  to  house  collections.  “Through  the  institution’s  ownership  or  guaranteed 
access,  sufficient  collections  and  services  are  readily  accessible  to  students  wherever 
programs  are  located  or  however  delivered.” 

7.3  Describes  the  financial  support  necessary  to  maintain  and  improve  library  and 
information  resources. 

7.4  Covers  the  type  of  staffing  and  the  provision  of  appropriate  orientation  and  training  for 
use  of  these  resources. 

7.4  Covers  the  type  of  staffing  and  the  provision  of  appropriate  orientation  and  training  for 
use  of  these  resources. 

7.6  Deals  with  evaluation  of  its  library  and  information  resources.  “The  institution 
regularly  and  systematically  evaluates  the  adequacy  and  utilization  of  its  library  and 
information  resources,  and  uses  the  results  of  the  data  to  improve  and  increase  the 
effectiveness  of  these  services.” 

Northwest  (1999) 

Eligibility  Criteria 

16... has  at  least  a core  library  and  learning  resources  appropriate  to  its  mission,  and  it 
provides  access  to  specialized  library  and  learning  resources  needed  for  independent  work 
in  the  fields  and  at  the  levels  represented  by  its  offerings. 

Standard  5.  Library  and  Information  Resources 

5.A  Purpose  and  Scope.  Three  statements  defining  the  components  of  information 
resources  and  services  (e.g.,  networks,  telecommunication  facilities,  computer  centers, 
programs.) 

5.B  Information  Resources  and  Services.  Five  statements  relating  to  equipment  and  the 
acquisition  and  organization  of  materials;  the  role  of  library  and  information 
resources/services  contributing  to  “developing  the  ability  of  students,  faculty,  and  staff  to 
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use  the  resources  independently  and  effectively;”  policies  and  procedures  for  the 
management  of  information  resources;  involvement  of  faculty  and  students  in  the  planning 
and  development  of  library  resources  and  services;  and  “5.B.5.  computing  and 
communications  services  are  used  to  extend  the  boundaries  in  obtaining  information  and 
data  from  other  sources  including  regional,  national  and  international  networks.” 

5.C.  Facilities  and  Access.  Two  statements  describing  the  need  for  adequate  facilities, 
collections  and  access.  “5.C.2.  In  cases  of  cooperative  relationships.. .formal  documented 
agreements  are  established.  These  cooperative  relationships  and  externally  provided 
information  sources  complement  rather  than  substitute  for  the  institution’s  own  adequate 
and  accessible  core  collection  and  services.” 

5.D.  Personnel  and  Management.  Six  statements  about  staffing,  professional  development, 
and  provision  of  “sufficient  financial  support  for  library  and  information  resources  and 
services.”  “5.D.4.  Organizational  arrangements  recognize  the  need  for  service  linkage 
among  complementary  resource  bases  (e.g.,  libraries,  computing  facilities,  instructional 
media,  and  telecommunication  centers).” 

5.E.  Planning  and  Evaluation.  Three  statements  describing  the  need  for  a planning  process 
that  involves  all  constituent  groups  and  “5.E.3.  The  institution  regularly  and  systematically 
evaluates  the  quality,  adequacy,  and  utilization  of  its  library  and  information  resources  and 
services,  including  those  provided  through  cooperative  arrangements,  and  at  all  locations 
where  courses  or  degrees  are  offered....” 

Supporting  Documentation  for  Standard  5 

Fourteen  “required  exhibits,”  including:  policies,  student  and  faculty  orientation  materials, 
use  statistics,  collection  statistics,  assessment  measures  to  determine  the  adequacy  of 
holdings  and  services  to  support  the  educational  programs  both  on  and  off  campus, 
computer  usage  statistics  related  to  retrieval  of  library  resources,  comprehensive  budgets 
for  library  and  information  resources. 

Standard  2.  Educational  Program 

also  has  two  relevant  statements: 

“2. A. 3.  Degree  and  certificate  programs  demonstrate  a coherent  design. ..synthesis  of 
learning,  and  the  assessment  of  learning  outcomes;  and  require  the  use  of  library  and  other 
information  resources.” 

“2.A.8.  Faculty,  in  partnership  with  library  and  information  resources  personnel,  ensure 
that  the  use  of  library  and  information  resources  is  integrated  into  the  learning  process.” 
subsection  “Graduate  Faculty  and  Related  Resources” 

“2.E.  1 . The  institution  provides  evidence  that  it  makes  available  for  graduate  programs  the 
required  resources  for  faculty,  facilities,  equipment,  laboratories,  library  and  information 
resources  wherever  the  graduate  programs  are  offered  and  however  delivered.” 

Standard  2G.  Continuing  Education  & Special  Learning  Activities 
“2.G.5.  Programs  and  courses  offered  through  electronically-mediated  or  other  distance 
delivery  systems  provide  ready  access  to  appropriate  learning  resources  and  provide 
sufficient  time  and  opportunities  for  students  to  interact  with  faculty.” 

Policy  on  Distance  Delivery  of  Courses,  Certificate,  & Degree  Programs 

contains  a “Library  and  Information  Resources”  section: 

“j.  The  institution  ensures  that  students  have  access  to  and  can  effectively  use  appropriate 
library  resources. 
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k.  The  institution  monitors  whether  students  make  appropriate  use  of  learning  resources. 

l.  The  institution  provides  laboratories,  facilities  and  equipment  appropriate  to  the  courses 
or  programs.” 

Southern  (1998) 

Standard  5.1.  Library  and  Other  Learning  Resources 

Very  prescriptive;  use  of  “must”;  fairly  detailed  about  specific  services  and  resources. 
Examples  follow. 

“5.1.1.  Purpose  and  Scope  of  Library  and  Learning  Resources.  The  library  and  other 
learning  resource  must  be  evaluated  regularly  and  systematically  to  ensure  that  they  are 
meeting  the  needs  of  their  users  and  are  supporting  the  programs  and  purpose  of  the 
institution. 

5.1.2.  Services.  Each  institution  must  ensure  that  all  students  and  faculty  have  access  to  a 
broad  range  of  learning  resources  at  both  primary  and  distance  learning  sites.  Basic  library 
services  must  include  an  orientation  program  designed  to  teach  new  users  how  to  access 
bibliographic  information  and  other  learning  resources.... Librarians  must  work 
cooperatively  with  faculty  and  other  providers  in  assisting  students  to  use  resource 
materials  effectively.... Convenient,  effective  access  to  electronic  bibliographic. databases, 
whether  on-site  or  remote,  must  be  provided  when  necessary  to  support  the  academic 
programs. 

5.1.3  Library  Collections.  Institutions  offering  graduate  work  must  provide  library 
resources  substantially  beyond  those  required  for  baccalaureate  programs. 

5.1.4  Information  Technology.  The  institution  must  provide  evidence  that  it  is 
incorporating  technological  advances  into  its  library  and  other  learning  resource 
operations. 

5.1.5  Cooperative  Agreements. 

5. 1.6  Staff. 

5.1.7  Library  and  Learning  Resources  for  Distance  Learning  Activities.  The  institution 
must  own  the  library/1  earning  resources,  provide  access  to  electronic  resources  available 
through  existing  technologies,  or  provide  them  through  formal  agreements.” 

Standard  5.3.  Information  Technology  Resources  and  Systems 

“5.3.  There  must  be  a reasonable  infusion  of  information  technology  into  the  curriculum  so 
that  students  exit  with  the  fundamental  knowledge  and  basic  ability  to  use  these  resource  in 
everyday  life.... A reliable  data  network  should  be  available  so  that  students,  faculty,  and 
staff  may  become  accustomed  to  electronic  communication  and  familiar  with  accessing 
national  and  global  information  resources.” 

Other  relevant  standards: 

Section  4:  Educational  Program.  4.3.  Graduate  Program 

“4.3.1.  An  institution  must  provide  a competent  and  productive  faculty,  adequate  library 
and  learning  resources,  adequate  computer  and  laboratory  facilities....” 

“4.3.4.  ...A  doctoral  degree  program  must  include  a substantial  period  of  residence  to 
provide  student  access  to  a wide  range  of  support  facilities,  including  a research  library, 
cultural  events....” 

Distance  Education:  Definition  And  Principles:  Policy  Statement 
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subsection  “Library  and  Learning  Resources”  (1997;  updated  May  2000) 
“The  institution  ensures  that  students  have  access  to  and  can  effectively  use  appropriate 
library  resources.  The  institution  monitors  whether  students  make  appropriate  use  of 
learning  resources.” 

Middle  States  (1994) 

Eligibility  Criteria 

“17.  The  institution  provides  sufficient  learning  and  information  resources  and  services  to 
support  the  nature,  scope  and  level  of  the  programs  offered.” 

Standard:  Library  and  Learning  Resources.  Access  and  Utilization 

Standards  are  not  numbered  and  are  in  prose  style,  but  the  paragraphs  can  be  summarized 
as  follows: 

1st  two  paragraphs  discuss  purpose,  goals,  and  scope  of  library  and  learning  resources. 

3rd  paragraph  describes  access  to  “a  broad  range  of  learning  resources  at  both  primary  and 
off-campus  sites....  A variety  of  contemporary  technologies  for  accessing  learning 
resources  and  instruction  in  their  use  should  be  available.” 

4th  paragraph  relates  to  “fostering  optimal  use  through  strategies  designed  to  help  students 
develop  information  literacy.” 

5th  paragraph  describes  access  to  remote  information  resources  via  ILL  and  resource 
sharing  or  through  formal  cooperative  agreements  and  networks. 

6th  paragraph  makes  it  clear  that  the  selection  of  materials  should  be  a collaborative  effort 
of  teaching  faculty,  librarians  and  other  information  providers. 

7th  paragraph  details  the  role  of  librarians  to  “demonstrate  their  professional  competence 
on  the  basis  of  criteria  comparable  to  those  for  other  faculty  and  staff.”  It  also  describes 
their  role  in  teaching  information  skills. 

8th  paragraph  describes  library  buildings  “in  making  an  attractive  place  for  study,  research 
and  teaching.  Nothing  else  matters  if  resources  are  not  used.” 

9th  paragraph  discusses  evaluation  of  all  learning  resources,  on-site  or  elsewhere.  “A 
system  for  assessing  the  effectiveness. ..should  focus  on  utilization,  accessibility, 
availability,  and  delivery  of  materials.  Quality  and  relevance  of  the  collections, 
effectiveness  of  reference  and  referral  services,  and  adequacy  of  funding  for  resources  and 
their  use  are  essential.  Ultimately,  the  most  important  measure  will  be  how  effectively 
students  are  prepared  to  become  independent,  self-directed  learners.” 

Educational  Program  and  Curricula 

7th  paragraph:  “Institutions  where  graduate  instruction  and  research  constitute  a major  part 
of  the  overall  program,  recognition  must  be  given  to  the  extended  demands  placed  on 
faculty,  finances,  facilities,  library  and  other  resources.” 

North  Central  (March 
2001) 

Eligibility  Criteria 

Educational  Program  18.  It  provides  its  students  access  to  those  learning  resources  and 
support  services  requisite  for  its  degree  programs. 

Standards  Criterion  2:  Human,  Financial,  and  Physical  Resources 

“j.  Academic  resources  and  equipment  (e.g.,  libraries,  electronic  services  and  products, 
learning  resource  centers,  laboratories  and  studios,  computers)  are  adequate  to  support  the 
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institution’s  purposes.” 

“Principles  of  Good  Practice  in  Adult  Degree  Completion  Programs” 

“Resources:  The  institution  ensures  access  to  learning  resources,  technology,  and  facilities 
to  support  its  adult  degree  completion  programs.” 

Southern  (March  2001 
draft) 

Eligibility  Criteria.  Core  Requirement  9 

“The  institution  provides  and  supports  student  and  faculty  access  and  user  privileges  to 
adequate  library  collections  as  well  as  to  other  leaming/information  technology  and 
information  resources  consistent  with  the  degrees  offered,  and  they  are  sufficient  to  support 
all  educational,  research,  and  public  service  programs.” 

Comprehensive  Standards.  Section  IV.  Programs,  subsection  “Library 
and  Other  Learning  Resources”  (one  of  the  four  “Programs”) 

“30.  The  institution  provides  facilities  and  instructional  support  services  for  the  library  and 
other  leaming/information  technology  resources  that  are  appropriate  to  its  courses  or 
programs  and  adequate  to  support  its  mission  and  its  effectiveness  for  learning. 

3 1 . The  institution  ensures  that  users  have  access  to  regular  and  timely  instruction  in  the 
use  of  the  library  and  other  learning/information  technology  and  information  resources. 

32.  The  institution  ensures  a sufficient  number  of  qualified  staff  with  appropriate  education 
or  experiences ...  to  accomplish  the  mission  of  the  institution.” 

Section  IV.  Programs,  subsection  “Educational  Programs” 

“14.  The  institution’s  use  of  technology  enhances  student  learning,  is  appropriate  for 
meeting  the  objectives  of  its  programs,  and  ensures  that  students  have  access  to  and 
training  in  the  use  of  technology.” 

WASC  (January  2001) 

Eligibility  Criteria 

“17.  The  institution  holds  or  otherwise  provides  long-term  access  to  sufficient  information 
and  learning  resources  to  support  its  purposes  and  all  of  its  educational  programs.  To 
supplement  these  resources  beyond  the  core  library  of  the  institution  there  may  be  specific 
long-term  written  arrangements  for  student  access  to  readily  available  resources.  Programs 
are  in  place  to  train  students  in  the  use  of  library  and  other  information  resources  and  to 
develop  information  literacy  skills.  The  institution  must  also  be  able  to  demonstrate  that 
library  use  is  a fundamental  part  of  curricula.” 

Standard  2.  Achieving  Educational  Objectives  through  Core  Functions, 
subsection  “Teaching  and  Learning” 

“Criterion  2.3.  The  institution’s  expectations  for  learning  and  student  attainment  are  clearly 
reflected  in  its  academic  programs  and  policies.  These  include  the  organization  and  content 
of  the  curricula,  its  admissions  and  graduation  policies,  the  organization  and  delivery  of 
advisement,  the  use  of  its  library  and  information  resources....” 

subsection  “Support  for  Student  Learning” 

“Criterion  2.13.  Student  support  services — including  financial  aid,  registration,  advising, 
career  counseling,  computer  labs,  and  library  and  information  services — are  designed  to 
meet  the  needs  of  the  specific  types  of  students  the  institution  serves  and  the  curricula  it 
offers.” 

23 


“Question  for  Institutional  Engagement 

To  what  extent  does  the  institution  provide  an  environment  that  is  actively  conducive  to 
study  and  learning,  where  library,  information  resources,  and  co-curricular  programs 
actively  support  student  learning?” 

Standard  3.  Developing  and  Applying  Resources  and  Organizational 
Structures  to  Ensure  Sustainability,  subsection  “Fiscal,  Physical,  and 
Information  Resources” 

“Criterion  3.6.  The  institution  holds  or  provides  access  to  information  resources  sufficient 
in  scope,  quality,  currency  and  kind  to  support  its  academic  offerings  and  the  scholarship 
of  it  members.  For  on-campus  and  students  enrolled  at  a distance,  physical  and  information 
resources,  services  and  information  technology  facilities  are  sufficient  in  scope  and  kind  to 
support  and  maintain  the  level  and  kind  of  education  offered.  These  resources,  services  and 
facilities  are  consistent  with  the  institution’s  purposes  and  are  appropriate,  sufficient,  and 
sustainable. 

Criterion  3.7.  The  institution’s  information  technology  resources  are  sufficiently 
coordinated  and  supported  to  fulfill  its  educational  purposes  and  to  provide  key  academic 
and  administrative  functions.” 

“Ouestion  for  Institutional  Engagement 

To  what  extent  do  the  institution’s  resources,  services  and  information  technology  respond 
to  faculty  needs  with  respect  to  scholarly  activity  and  curricular  development?” 

Middle  States 
(February  2001  draft) 

Eligibility  Criteria 

“17.  The  institution  provides  sufficient  learning  and  information  resources  and  services  to 
support  the  nature,  scope  and  level  of  the  programs  offered.” 

Standard  4.  Institutional  Resources 

“An  accredited  institution  is  characterized  by  recognition  in  the  comprehensive  plan  that 
facilities,  such  as  libraries  and  other  learning  resources,  which  are  fundamental  to  all 
educational  and  research  programs,  are  adequately  supported  and  staffed  to  accomplish  the 
institution’s  objectives  for  student  learning,  both  on  campuses  and  at  a distance.” 

Standard  10.  Educational  Programs 

“Fundamental  Elements:  learning  resources,  library  services,  and  professional  library  staff 
support  adequate  to  support  the  institution’s  educational  programs;  collaboration  between 
professional  library  staff  and  faculty  in  teaching  and  fostering  information  literacy  skills 
relevant  to  the  curriculum  and  to  faculty  research;  programs  that  promote  student  use  of 
information  and  learning  resources.” 

Standard  12.  Branch  Campuses,  Additional  Locations  & Other 
Instructional  Sites 

“Supplemental  Analysis  and  Documentation:  analysis  of  the  adequacy  and  appropriateness 
of  library/information  and  other  learning  resources.” 

Standard  12.  Specialized  Educational  Activities — Distance  or 
Distributed  Learning 

“Fundamental  Elements:  available,  accessible,  and  adequate  learning  resources  (such  as 
libraries  or  other  information  resources)  appropriate  to  the  programs  offered  at  a distance.” 
“Supplemental  Analysis  and  Documentation:  evidence  of  how  the  institution  assures  that 
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students  and  faculty  have  sufficient  technological  and  information  literacy  skills  to  access 
and  use  effectively  the  information  resources  available  at  a distance.” 
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